Case change breaks builds?

[lonelycode]

Not sure what's started to happen, but overnight our builds have started to break:

can't load package: package github.com/TykTechnologies/tyk: case-insensitive import collision: "github.com/Sirupsen/logrus" and "github.com/sirupsen/logrus"

I wonder if it's the case change from S to s in the github username of the account. Even if we change the import path to use the lowercase version, dependencies that use the uppercase version that are not under our control seem to need updating too.

Vendoring the lib doesn;t seem to work either, as the log.WithFields seems to self referential, but maybe I was too hasty in dismissing vendoring:

./api.go:124: cannot use "github.com/TykTechnologies/tyk/vendor/github.com/Sirupsen/logrus".Fields literal (type "github.com/TykTechnologies/tyk/vendor/github.com/Sirupsen/logrus".Fields) as type "github.com/Sirupsen/logrus".Fields in argument to log.WithFields

Any tips appreciated.

[sdboyer]

@stevvooe ok, i'm a bit confused at this point, as it seems like you're trying to convince me to do something that we're already doing. maybe we're just talking past each other somehow. because:

What is the point in allowing the symbol golang.docker.io/docker/vendor/github.com/Sirupsen/logrus.Errorf if introducing golang.docker.io/docker/vendor/github.com/sirupsen/logrus.Errorf just causes a compiler error?

yes. agreed. 💯. and disallowing that case is exactly what we're doing, because it causes a compiler error. i've said or alluded to as much in every one of my last three comments, and the links i've already provided add more detail.

[stevvooe]

@sdboyer My question has been asking about a whether or not this should be a fix in the Go toolchain (see https://github.com/sirupsen/logrus/issues/451#issuecomment-326649038). There is no point in making the dep tool, go get and GOPATH support case-sensitive import paths when the compiler explicitly disallows it.

Packages with the same name but different casing need to be treated as the same package. While golang/dep#1079, it is unrealistic to have the solver look for solutions that will select a variant of github.com/Sdboyer/deptest based on what is imported. In reality, I want this to use whatever package is canonically folded to github.com/sdboyer/deptest. There is no use case for having multiple instances, even of they all agree on the same import casing.

[sdboyer]

My question has been asking about a whether or not this should be a fix in the Go toolchain

yes, i saw that, but that doesn't make sense. the go toolchain doesn't do any kind of version selection right now. that's the problem we're fixing in dep. there is no conceivable way of "fixing" this in the go toolchain, apart from moving the responsibility for version selection into the go toolchain, which is the point of dep in the first place.

There is no point in making the dep tool, go get and GOPATH support case-sensitive import paths when the compiler explicitly disallows it.

i don't know why you think anyone is proposing this. i'm not. the PRs aren't. the PRs here REMOVE case sensitivity from (portions of) the import path, in a way that brings them into line with the compiler.

Packages with the same name but different casing need to be treated as the same package.

yes. agreed. and that's what the changes to dep in that PR are enforcing.

it is unrealistic

i'm not sure what this means. the PR literally works, so it's "realistic" in that sense. and it adds no new crazy logic in the solving algorithm - just one more dimension of satisfiability, wherein we verify that all projects active in the depgraph agree on a case variant for all project roots. now, that's not nothing - it's bumping the constant factor on an NP-hard search algorithm - but adding one more dimension doesn't just push us from "realistic" to "unrealistic."

In reality, I want this to use whatever package is canonically folded to github.com/sdboyer/deptest. There is no use case for having multiple instances, even of they all agree on the same import casing.

so, wait - you're saying you want the system (the compiler, dep, etc.) to enforce that import paths must be written in their case-folded form? if so, then OK, we do have a disagreement, at minimum because:

1. that would render large swathes of existing Go code invalid
2. it would require the compiler to be able to differentiate between "root" (e.g. github.com/foo/bar) and "package" portions of import paths, at minimum because the former are governed by hosting platforms and the latter by code authors. doing that is not always possible without network access for go-get metadata, and that's a place the compiler can't go.

but maybe i'm still not understanding, and that's not what you mean?

[stevvooe]

@sdboyer

yes, i saw that, but that doesn't make sense. the go toolchain doesn't do any kind of version selection right now. that's the problem we're fixing in dep. there is no conceivable way of "fixing" this in the go toolchain, apart from moving the responsibility for version selection into the go toolchain, which is the point of dep in the first place.

No one said we should fix this with versioning. The problem is that the compiler enforces that only a single casing of a package import exists. I am not sure why you think there is no conceivable way of fixing this in Go. Do you have an analysis showing this? Go does import resolution within GOPATH and there may be a possible fix there that doesn't involve a versioning system.

The issue right now is a project and all of its imports need to have the same casing for a package import. If the compiler allowed mixed casing and only raised an error when the package is ambiguous (two casing versions are encountered at the same import priority), we could correct the problem by modifying the set of imports (for example, make sure that only github.com/sdboyer/deptest and not github.com/Sdboyer/deptest), rather than having all of the imported projects agree on a single import path.

Case-folding import paths would not have an effect existing Go code. No one expects github.com/sdboyer/deptest and github.com/Sdboyer/deptest to be different packages. Adding this as a constant factor in the SAT solver seems like a waste, when the package name could be canonicalized to github.com/sdboyer/deptest before adding the case variations to the solution space. When I say this is unrealistic, it means that it doesn't actually help with the problem; all packages have to agree on a single casing of import paths. I am arguing that this shouldn't be a requirement in the first place and making this a requirement is what makes the problem untenable.

Put simply, there may be a solution that exists by moving the enforcement of case variations to the package resolution stage, rather than based on encountered imports. It would then be the problem of dep or some other dependency system (GOPATH, vendor, etc.) to ensure that canonicalized import paths are unique, which is a solvable problem.

[sdboyer]

I am not sure why you think there is no conceivable way of fixing this in Go. Do you have an analysis showing this?

ahhh, i think i finally see what you're arguing for here. at least, the general shape of it.

my impossibility claim there was based on the assumption that you were talking about doing this via version selection. now that i better understand what you're saying, though, sure - i suspect it's quite possible to do what you're suggesting. it's not clear to me whether it's the right path, though.

two casing versions are encountered at the same import priority

i'm not sure what "import priority" is, but "priority" suggests some concept of ordinality. while we can impose something like that on a DAG with a "shortest/longest path distance from root", it's not clear to me that that's actually a useful distinction for this domain in general, or for this problem in particular. hard to tell though, because:

we could correct the problem by modifying the set of imports (for example, make sure that only github.com/sdboyer/deptest and not github.com/Sdboyer/deptest)

this seems like an incomplete thought? i don't know what the "modification" here is.

making this a requirement is what makes the problem untenable.

at most, i think we can say it makes the problem frustrating. if it were so critically flawed an approach as to be unrealistic, then...well, a bit facetious, but we'd hear about it in TIOBE analyses about Go's viability. what we know is that the check exists in the compiler today, and the actual felt pain from it here, as reflected in this issue, seems to be a confluence of two factors:

1. logrus is either the or one of the most widely imported public package in the Go ecosystem
2. Sirupsen chose to change the casing of his name, and people followed suit with their imports

it is, of course, not impossible that this might happen again in the future. but it strikes me as more of an edge cases, having required a bit of a perfect storm to really have bite.

Adding this as a constant factor in the SAT solver seems like a waste, when the package name could be canonicalized to github.com/sdboyer/deptest before adding the case variations to the solution space.

so the proposal is that the compiler only ever sees the folded/canonicalized paths, and that's the information it has when it goes searching on local disk for files to satisfy an import?

Adding this as a constant factor in the SAT solver seems like a waste, when the package name could be canonicalized to github.com/sdboyer/deptest before adding the case variations to the solution space.

so, to be clear, as a practical matter, this has to happen in dep right now, as it's the only solution we have to fix the pain that exists with currently released versions of the toolchain. we can consider taking it out later if a change like you're proposing lands, but it'd probably have to be quite far in the future. dep has to care about a longer span of time than your average project.

[stevvooe]

@sdboyer That is a closer interpretation of what I'm saying.

I think the problem with the dep solution is that it won't actually be able to solve the problem if dependencies don't agree on case. The situation is no better or worse than it is today, except we might get a better error message.

Either way, let's shift venues. I'm not sure this is the right place to discuss. I haven't found an issue on golang for this, but it is likely that one exists.