cmd/go: allow replacement modules to alias other active modules

mwf commented 6 years ago

What version of Go are you using (`go version`)?

Go tip: go version devel +f2131f6e0c Wed Aug 8 21:37:36 2018 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes (it is not reproduced with go version go1.11beta2 darwin/amd64)

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/ikorolev/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/var/folders/_b/d1934m9s587_8t_6ngv3hnc00000gp/T/tmp.cqU8g8OM/gopath"
GOPROXY=""
GORACE=""
GOROOT="/Users/ikorolev/.gvm/gos/go1.11beta3"
GOTMPDIR=""
GOTOOLDIR="/Users/ikorolev/.gvm/gos/go1.11beta3/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/var/folders/_b/d1934m9s587_8t_6ngv3hnc00000gp/T/tmp.cqU8g8OM/vgo-a-user/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_b/d1934m9s587_8t_6ngv3hnc00000gp/T/go-build138999780=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Sorry, no standalone reproduction, since issue is connected with repository forking

Assume we have a repository A: https://github.com/mwf/vgo-a with the only feature:

package a

var A = "A"

Than we have a fork1 https://github.com/mwf/vgo-a-fork1, adding a feature B :

package a

var B = "B is a new feature in a-fork1"

Unfortunately fork1 will never be merged to the upstream, just because a author don't like this feature.

It's important to note, that both a and a-fork1 don't have go.mod, they are too conservative for that 😄

Then we got a happy user, using both projects in his repo. go.mod:

module github.com/mwf/vgo-a-user

require (
    github.com/mwf/vgo-a v0.1.0
    github.com/mwf/vgo-a-fork1 v0.2.0
)

main.go

package main

import (
    "fmt"

    "github.com/mwf/vgo-a"
    a_fork "github.com/mwf/vgo-a-fork1"
)

func main() {
    fmt.Printf("A: %q\n", a.A)
    fmt.Printf("B: %q\n", a_fork.B)
}

All just works fine:

$ go run .
A: "A"
B: "B is a new feature in a-fork1"

Here appears fork2 https://github.com/mwf/vgo-a-fork2, forked from fork1, and fixing some bugs both in the upstream and in fork1.

We use the fork2 with replace in our main repo: https://github.com/mwf/vgo-a-user/blob/master/go.mod

module github.com/mwf/vgo-a-user

require (
    github.com/mwf/vgo-a v0.1.0
    github.com/mwf/vgo-a-fork1 v0.2.0
)

replace github.com/mwf/vgo-a => github.com/mwf/vgo-a-fork2 v0.2.1

replace github.com/mwf/vgo-a-fork1 => github.com/mwf/vgo-a-fork2 v0.2.1

What did you expect to see?

Building this with go1.11beta2 works just fine:

cd `mktemp -d`
git clone git@github.com:mwf/vgo-a-user.git .
go version && go run .

Output:

go version go1.11beta2 darwin/amd64
go: finding github.com/mwf/vgo-a-fork2 v0.2.1
go: downloading github.com/mwf/vgo-a-fork2 v0.2.1
go: finding github.com/mwf/vgo-a v0.1.0
go: finding github.com/mwf/vgo-a-fork1 v0.2.0
A: "A, fixed in a-fork2"
B: "B, fixed in a-fork2"

What did you see instead?

Building with the tip (and beta3) returns an error:

cd `mktemp -d`
git clone git@github.com:mwf/vgo-a-user.git .
go version && go run .

Output:

go version devel +f2131f6e0c Wed Aug 8 21:37:36 2018 +0000 darwin/amd64
go: finding github.com/mwf/vgo-a-fork2 v0.2.1
go: downloading github.com/mwf/vgo-a-fork2 v0.2.1
go: github.com/mwf/vgo-a-fork1@v0.2.0 used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

More comments

I understand that this case is very specific and arguable - this should not ever happen ideally, but we have the real case here: https://github.com/utrack/clay/blob/master/integration/binding_with_body_and_response/go.mod

There is a little workaround, to define go.mod at fork2 and make a replace upstream -> fork2_with_go.mod, but it's too dirty :)

replace github.com/mwf/vgo-a => github.com/mwf/vgo-a-fork2 v0.3.0 // version with go.mod
replace github.com/mwf/vgo-a-fork1 => github.com/mwf/vgo-a-fork2 v0.2.1 // no go.mod

It works with tip and beta3:

$ go version && go run .
go version devel +f2131f6e0c Wed Aug 8 21:37:36 2018 +0000 darwin/amd64
A: "A, fixed in a-fork2"
B: "B, fixed in a-fork2"

If you decide that the case is too specific and crazy, and you'd like to close as "Won't fix" - then I assume we should change the error string, because it's confusing now:

go: github.com/mwf/vgo-a-fork1@v0.2.0 used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

It should look like this:

go: github.com/mwf/vgo-a-fork2@v0.2.1 used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

because it's github.com/mwf/vgo-a-fork2 who's to blame for the error.

mwf commented 6 years ago

And this crazy stuff could happen only in non-go.mod repositories, because you can't use both upstream and fork at the same time if they have go.mod inside:

module github.com/mwf/vgo-a-user

require (
    github.com/mwf/vgo-a-mod v0.1.0
    github.com/mwf/vgo-a-mod-fork1 v0.2.0
)

$ go run .
go: finding github.com/mwf/vgo-a-mod-fork1 v0.2.0
go: github.com/mwf/vgo-a-mod-fork1@v0.2.0: parsing go.mod: unexpected module path "github.com/mwf/vgo-a-mod"
go: finding github.com/mwf/vgo-a-mod v0.1.0
go: error loading module requirements

You are forced either to use your fork as replace argument, or to change the module in fork's go.mod and never assume it as a fork again :) And this is for the best 👍

So no one will ever hit such an issue in the brave new world of go modules 😄

bcmills commented 6 years ago

Ideally, I think the long-term solution will be to treat replacements as rewriting the import paths rather than the source code, to allow for precisely this kind of fork-unification behavior.

(For an example of how this can go wrong otherwise, see the code in #26607.)

mwf commented 6 years ago

@bcmills you set "Go1.12" milestone for the issue.

Maybe we could at least fix the error string in Go1.11 to eliminate the confusion?

If you decide that the case is too specific and crazy, and you'd like to close as "Won't fix" - then I assume we should change the error string, because it's confusing now:

go: github.com/mwf/vgo-a-fork1@v0.2.0 used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

It should look like this:

go: github.com/mwf/vgo-a-fork2@v0.2.1 used for two different module paths (github.com/mwf/vgo-a and github.com/mwf/vgo-a-fork1)

because it's github.com/mwf/vgo-a-fork2 who's to blame for the error.

bcmills commented 6 years ago

Oh, yeah, that's an easy fix.

gopherbot commented 6 years ago

Change https://golang.org/cl/128878 mentions this issue: cmd/go/internal/modload: emit correct module in duplication error

F21 commented 5 years ago

I am also seeing this problem when using go mod why. Since testcontainers/testcontainer-go was renamed to testcontainers/testcontainers-go recently, modules break for me since it is being used by one of my dependencies (not sure how deep).

I added:

replace github.com/testcontainers/testcontainer-go => github.com/testcontainers/testcontainers-go v0.0.0-20190108154635-47c0da630f72

to go.mod in order to get go get -u and go mod tidy to work correctly.

However, if I run go mod why github.com/testcontainers/testcontainers-go, I get:

go: finding github.com/testcontainers/testcontainers-go latest
go: github.com/testcontainers/testcontainers-go@v0.0.0-20190108154635-47c0da630f72 used for two different module paths (github.com/testcontainers/testcontainer-go and github.com/testcontainers/testcontainers-go)

bcmills commented 5 years ago

We should consider how to address #30831 in concert with this change. (Replacement aliasing seems clearly “necessary” for #30831, but we should make it as close as possible to “sufficient” too.)

gopherbot commented 5 years ago

Change https://golang.org/cl/174939 mentions this issue: cmd/go: use replacement, not replaced, module paths in binary

Smityz commented 5 years ago

I am confusing the situation,is there any solution can fix my problem(#32462 )? I am sorry,i am a rookie in programming.

bcmills commented 5 years ago

@Smityz, it's not clear why you need those replace statements to begin with. If you need help setting up a project using modules (and have already checked http://golang.org/wiki/Modules for solutions), the venues on http://golang.org/wiki/Questions are likely to be more effective.

Smityz commented 5 years ago

@Smityz, it's not clear why you need those replace statements to begin with. If you need help setting up a project using modules (and have already checked http://golang.org/wiki/Modules for solutions), the venues on http://golang.org/wiki/Questions are likely to be more effective.

In China.For some reasons,we can't access golang.org or other repository,the only repository we can access is which in github,so I have to do that to get my package.

agnivade commented 5 years ago

Use go mod edit -replace=<src>=<target> to replace golang.org modules by github.com paths. See https://github.com/golang/go/issues/28652 for more examples and context. Use http://golang.org/wiki/Questions if you have further questions.

bcmills commented 5 years ago

replace in general needs some rework, which we plan to investigate in conjunction with broader improvements to the MVS / module loading process.

The right syntax for this sort of replace depends on what we can (or can't) eliminate at that point, so putting on hold until 1.15 so that we can tackle all of these holistically.

mwf commented 4 years ago

Here is one more example for further investigation.

Half of the company still uses dep for vendoring, but we've got migration to modules in progress.

Since dep knows nothing about /vX library notation (https://github.com/golang/dep/issues/1962), we can't switch to versioned import paths, but in order to use modules we have to use replace directives.

Imagine we have library A, library B and service S. Libraries are used both in dep-based and modules-based projects.

Library A go.mod:

module go.avito.ru/gl/A/v7

...

Library B go.mod looks like:

module go.avito.ru/gl/B

require(
   go.avito.ru/gl/A v7.1.0+incompatible // we need this in order to preserve dep compatibility in Gopkg.toml
)

replace go.avito.ru/gl/A => go.avito.ru/gl/A/v7 v7.1.0

And service S has

module go.avito.ru/av/S

require (
    go.avito.ru/gl/A/v7 v7.1.0 // direct dependency on module-based A lib
)

replace go.avito.ru/gl/A => go.avito.ru/gl/A/v7 v7.1.0

The service S build ends up with error:

$ go build ./cmd/service
go: go.avito.ru/gl/A/v7@v7.1.0 used for two different module paths (go.avito.ru/gl/A and go.avito.ru/gl/A/v7)

What helps us - go mod vendor somehow bypasses this error (have you made it intentionally?) and vendors go.avito.ru/gl/A under both passes in vendor tree. Thus go build -mod=vendor ./cmd/service goes without any error.

PS I believe this problem is much more of real-world than the original one in my issue. So if you like, I could edit the original one in favour of an updated case. Or better just leave it as is?

bcmills commented 4 years ago

@mwf, note that the modules using the major-subdirectory layout should work with versioned import paths with both dep and modules.

A replace directive should not be necessary, especially since that forces all downstream consumers of your module to replicated that directive in their own go.mod file in order to successfully build.

mwf commented 4 years ago

Yeap, thanks!

We use simple flow with latest major in master branch. I'm not sure if we could switch to subdirectory layout for now.

dmitshur commented 4 years ago

This issue is currently labeled as early-in-cycle for Go 1.15. That time is now, so friendly ping. If it no longer needs to be done early in cycle, that label can be removed.

kolaente commented 4 years ago

I'm hitting this with a depdendency of a dependency of a dependeny of a dependency (with the exact dependency from #37911). The interesting thing is that none of the packages used in my application actually use the faulty package. They all actively use the fixed version, but have a reference to the broken one in their go.sum file (the go.mod file is fine everywhere).

Edit: I was able to """fix""" that by forking the module, changing its path and then adding a replace to my go.mod to use the fork with the change package name. While this works, it is very dirty and should not be necessary.

kevinburke1 commented 4 years ago

Just a heads up that one of the linked answers is the top result on Google when you search for "used for two different module paths" so it might be good to provide some easy guidance here on how to solve that problem. As it stands you have to read a lot to get your head around how this solves the issue.

dmitshur commented 4 years ago

This issue is currently labeled as early-in-cycle for Go 1.16. That time is now, so this is a friendly ping so the issue is looked at again.

bcmills commented 4 years ago

I'm dropping the early-in-cycle, label on the theory that (empirically) hardly anybody tests the more advanced details of module mode until at least the beta anyway.

mvdan commented 4 years ago

To add another real case where this is causing pain: github.com/gopherjs/vecty has been moved to github.com/hexops/vecty, but dozens of library modules like github.com/marwan-at-work/vecty-router still require the old module. I tried upgrading vecty to master and then using replace github.com/gopherjs/vecty => github.com/hexops/vecty master to force third party libraries to come along for the upgrade, but encountered this issue.

cc @marwan-at-work in case he's open to fixing this in his module directly.

marwan-at-work commented 4 years ago

@mvdan done on the vecty-router side thanks to @NateWilliams2 https://github.com/marwan-at-work/vecty-router/pull/12

iamkirkbater commented 4 years ago

Another case where this is causing pain:

An upstream dependency (A) is pointing to a git repo that has been deleted (B). Another dependency (C) is using a fork (D) of the repo that has been deleted. The fork (D) and the repo that has been deleted (B) are both exactly the same, just B no longer exists.

So a simplified Dep tree would look like this:

Our Project
  | - Dependency
  |   | - Dependency A
  |   |   | - Package B (now deleted)
  | - Dependency C
  |   | - Package D (Fork of B with no changes)

As a stopgap to continue our builds until the upstream dependency (A), we thought we could just replace B with D, but that's throwing the "X used for two different module paths", even though that's exactly what we want to do.

Dragomir-Ivanov commented 4 years ago

I have another issue with 1.15.3. Maybe I need to file another Issue, but will post here for now: Having these in go.mod

replace gopkg.in/mgo.v2 => ../mgo

replace github.com/globalsign/mgo => ../mgo

Results in: go: ../mgo@ used for two different module paths (github.com/globalsign/mgo and gopkg.in/mgo.v2). I feel that this should be allowed.

bcmills commented 4 years ago

@Dragomir-Ivanov, no need to file a separate issue. This is next on my list after #36460, although it might not make 1.16 before the freeze.

However, note that github.com/globalsign/mgo imports packages from itself. The replace directives that you describe, if they were allowed today, would result in two distinct copies of each package (one beginning with github.com/globalsign/mgo, and one beginning with gopkg.in/mgo.v2), but with each copy importing only one other copy (presumably github.com/globalsign/mgo?).

That could well produce type errors if types from the imported packages are used in the exported API, since each distinct package also defines its own types.

It seems that what you ideally need is exactly what we haven't implemented yet, which is a way to reinterpret import statements that refer to one of those copies as if they instead referred to the other.

Dragomir-Ivanov commented 4 years ago

@bcmills Thanks for your reply and all the efforts! I workaround my issues so far, by making private fork, and replaced all self import paths in github.com/globalsign/mgo to gopkg.in/mgo.v2. In any case, I really think that once you replace a package, go toolchain should implicitly redirect replaced package self-imports to the replacer.

There are lots of abandoned packages, then revived by someone else and continued (as this mgo one), and we need a clean way to replace the package in the whole project(thus replacing package in our dependencies as well) without any forks, source modifications, etc.

I am in no rush what so ever, and can wait for 1.16, 1.17, etc. Also I am thinking in what takes if we want to replace a package only for certain dependency packages. This can quickly become NPM :)

fwhezfwhez commented 3 years ago

go 1.12.17

used for two different module paths for github.com/golang/protobuf and golang.org/x/protobuf.

To fix this, I replace all my import path of 'golang.org/x/protobuf' to 'github.com/golang/protobuf'.

Sincerely I think these two import pkgs should support work together and not just force user to abandon one of it, because everything work fine in GOPATH and just go bad in GOMODULE. It's just not that smooth.

Rikanishu commented 3 years ago

Can anyone explain me why it's not possible to make go point to the same version of package from multiple aliases? I don't understand the reason why a compilation process exits with the error. Looks like it's an inner go mod problem, not a something that should cause fail by design.

Will it be resolved somehow? Any progress of that?

Thanks!

bcmills commented 3 years ago

@Rikanishu, #26607 is why aliasing is currently disallowed. I plan to address this in Go 1.17.

fwhezfwhez commented 3 years ago

Can anyone explain me why it's not possible to make go point to the same version of package from multiple aliases? I don't understand the reason why a compilation process exits with the error. Looks like it's an inner go mod problem, not a something that should cause fail by design.

Will it be resolved somehow? Any progress of that?

Thanks!

Maybe go chain will only find pkg source code by one and the only one specific key, this key is module-name.

Why two pkg sharing a same module-name will conflict. It(go chain) might use a map to storage a source code ref like:

refs map[string]*moduleRef

key is module-name, value is where source code put.

Here it meet a question!

When analyse to import golang.org/x/protobuf, although it imported-path is named like this but its module name is github.com/golang/protobuf.Thus, it will be temporary stored as

if _, exist := refs["github.com/golang/protobuf"]; !exist  {
refs["github.com/golang/protobuf"] = newModuleRef("GOPATH/pkg/mod/golang.org/x/protobuf")   // ✔
} else {
panic("used for two different module path")
}

When analyise to import github.com/golang/protobuf, its module name is also github.com/golang/protobuf. Then do a same operation

if _, exist := refs["github.com/golang/protobuf"]; !exist  {
refs["github.com/golang/protobuf"] = newModuleRef("GOPATH/pkg/mod/github.com/golang/protobuf")   
} else {
panic("used for two different module path")     // ✖
}

So only extend refs key format, this question can be solve then.

ianlancetaylor commented 3 years ago

Moving to Backlog.

bcmills commented 3 years ago

@ianlancetaylor, I still intend to address this issue as soon as I finish implementing #36460. I'm not sure whether this issue will make 1.17 or wait for 1.18, but it's not Backlog.

ianlancetaylor commented 3 years ago

We may have a conflict with the release milestones. All issues with a milestone of Go1.17 have to be resolved for the 1.17 release, where the resolution is often to kick the milestone forward. Kicking forward to Go1.18 leads to pointlessly examining issues and kicking them forward for every release. So we introduced the Backlog milestone to mean "we want to do this, but we don't know exactly when." Basically, every issue with a Go1.17 milestone is a cost that is paid several times over as people keep looking at the issue again and again.

When you say that the issue will be in 1.17 or 1.18, what that means to me is that either 1) the issue could be in 1.17 but must be in 1.18, in which case the milestone should be Go1.18, or 2) the issue should be done but the exact release doesn't matter, in which case the milestone should be Backlog.

It sounds like you are using the milestone to mean something slightly different. Perhaps that could be conveyed by labels or the assignee field?

Basically I'm trying to reduce load on the release team, including myself. There are nearly 200 issues in the Go1.17 milestone. Looking at each of those is surprisingly time consuming.

Thanks.

bcmills commented 3 years ago

When you say that the issue will be in 1.17 or 1.18, what that means to me is that either 1) the issue could be in 1.17 but must be in 1.18, in which case the milestone should be Go1.18, or 2) the issue should be done but the exact release doesn't matter, in which case the milestone should be Backlog. … It sounds like you are using the milestone to mean something slightly different. Perhaps that could be conveyed by labels or the assignee field?

I am having trouble rectifying that description with the milestones as applied to other issues. We already convey the information about whether a milestone should block the release using the release-blocker label, and when I have questioned releases cut without auditing issues (especially backport issues) milestoned to that release, the release team has been adamant that if I want to ensure that an issue is even reexamined prior to a release it must be labeled release-blocker. This issue is not labeled release-blocker, and the 1.17 freeze is still a week and a half out.

If we want a milestone by itself to mean “this issue will be re-examined by the release team before the milestone release is cut”, then we should apply that policy uniformly, not just to the major-release milestones.

For the moment I will move this to the 1.18 milestone, since it does not need to be reexamined by the release team before the 1.17 release.

ianlancetaylor commented 3 years ago

Yes, the release-blocker label is required to ensure that the issue is examined. But a milestone without the release-blocker label still means something.

I agree that major and minor release milestones are treated differently, and I don't see a problem with that. It doesn't make sense to use a minor release milestone without the release-blocker label. At least, I don't know what that would mean. But of course it does make sense to use a major release milestone without release-blocker, and people do that all the time.

Let me turn it around: what would you like the Go1.17 milestone to mean? How would it help you to use that milestone?

bcmills commented 3 years ago

@DocMerlin mentioned an interesting use-case for path-rewriting replacements.

Suppose that a module previously existed at a path like github.com/foo/bar, and contained a package like github.com/foo/bar/baz/bag. Then the author moved the package out to its own repo, github.com/foo/bag. A user might want to replace github.com/foo/bar/baz/bag with github.com/foo/bag, even though the former path is not really a module at all.

To make matters worse, if the author of github.com/foo/bar didn't actually remove the old ./baz/bag package (perhaps the repo is archived and unmaintained!), then just injecting a new module at github.com/foo/bar/baz/bag would produce an ambiguous import error instead of replacing the packages as intended.

gopherbot commented 3 years ago

Change https://golang.org/cl/332571 mentions this issue: cmd/go/internal/modload: remove ImportMap and PackageDir

bcmills commented 3 years ago

With everything else happening in Go 1.18, I think this is going to slip to 1.19.

tandr commented 3 years ago

Looks like there are more than one thing slipping to 1.19 because of the same reasons. May I suggest to make 1.18 with all these items, and then make 1.19 to go with generics (maybe shortly after)?

ianlancetaylor commented 3 years ago

@tandr That could have been a good discussion six months ago, but at this point we only have one month left to the 1.18 feature freeze. We're not going to be able to significantly rearrange our priorities now. Sorry.

tandr commented 3 years ago

Thanks Ian, I understand.

integrii commented 2 years ago

I just want to leave a note here to say that I've spent the last two hours trying to get a package in the github.com/docker/docker module to build, but can't because of this fix not being in yet.

go: github.com/sirupsen/logrus@v1.8.1 used for two different module paths (github.com/Sirupsen/logrus and github.com/sirupsen/logrus)

Specifically, the replace line below for go.mod below is not possible in go 1.18, and that means I can not get my build to work no matter what I do.

replace github.com/Sirupsen/logrus v1.8.1 => github.com/sirupsen/logrus v1.8.1

ianlancetaylor commented 2 years ago

@bcmills @matloob This issue is marked for 1.19. It's just been rolling forward in milestones. Should it move to Backlog?

bcmills commented 2 years ago

We've been doing active design work for this issue this week. It's not ready for the Backlog quite yet.

ianlancetaylor commented 4 months ago

This is still open and just rolling forward. @matloob should we move this to backlog? Thanks.

matloob commented 4 months ago

Yes, let's move this to backlog. We would like to fix this issue at one point but it's unlikely to be done in 1.24.

golang / go