Open bozaro opened 2 years ago
Tidiness is a property of an individual module, not a workspace: if a module is tidy, then a downstream consumer of the module knows which versions to use for every dependency of every package in that module.
If you don't particularly care about downstream consumers having a package that is provided by the workspace, you can use go mod tidy -e
to ignore the error from the missing package.
Otherwise, you either need to publish the workspace dependencies before running go mod tidy
(so that they have well-defined upstream versions), or tell the go
command what those versions will be (using a replace
and require
directive in the individual module).
@matloob, for Go 1.19 I wonder if we should augment the go.work
file to be able to declare explicit intended versions for the modules in the workspace. Then go mod tidy
could use those versions instead of looking upstream. 🤔
Currently with reproducing repository (https://github.com/bozaro/go-work-play/tree/go-mod-tidy).
I can run:
cd tools/hello
go run .
And go run
works like workspace go modules declared as replace
in current tools/go.mod
.
But for go mod tidy
I need add replace
manually.
This behaviour looks like inconsistent: I expects that go run
and go mod tidy
would use same dependencies.
What valid use cases are there for go.work
if we are still required to fully implement the go.mod
in order to work on multi module repo?
@bcmills
This is also happening for go get ./...
I agree with @liadmord, the only benefit I see get from the go.work
is a better "understanding" of my IDE
Or - maybe I'm doing something wrong here?
I agree with @bozaro @liadmord and @AlmogBaku
I have a monorepo private project where I have multiple modules, and on each one I need to insert a lot of replace
statements for each direct and indirect dependencies. With go workspaces
I really want to be possible to run go mod download
or go mod tidy
without the needed to write a replace statement in every go.mod
file in my workspace.
Many of my colleagues have this need
a big problem for go 1.18 workspace mode
Encountering the same issue with go1.18.2
.
Furthermore, some other go mod
commands also failed. See sample repo: https://github.com/maxsxu/go-workspace
go mod vendor
➜ server go mod vendor
atmax.io/go-workspace/server imports
atmax.io/go-workspace/commons/pkg: no required module provides package atmax.io/go-workspace/commons/pkg; to add it:
go get atmax.io/go-workspace/commons/pkg
go mod verify
➜ server go mod verify
atmax.io/go-workspace/server : missing ziphash: open hash: no such file or directory
Expected behaviour: The go mod
commands should not try to find modules which have been declared in go.work
. which I think will make go workspace more meaningful.
My expected functionality is that a go mod tidy
on a go.work
enabled workspace should feel like a bunch of phantom replace
statements inside the go.mod
without editing the file. If the module is at github.com/Gbps/mymodule
and I tell my go.work
that the module can be found in /path/to/mymodule
, then go mod tidy
should pull the version from my local repo instead of going to github at all.
If I don't commit my go.work
, then anyone who uses my go.mod
afterwards will have to go to GitHub, which is correct! Now I can edit my local copy and test changes, but when I'm ready to push, I can tidy
the correct version into the go.mod
and push both the mymodule
local dev repo and my current project. This makes sense!
Once I must make an edit to go.mod
to include a local path string for any feature of the go mod
command line to work, all the purpose of go.work
is lost on me. I should never have to taint go.mod
with local paths, go mod tidy
can find what commit I'm working with locally through go.work
and update it accordingly. It should be no different than if I pushed a commit to github and ran go mod tidy
to pull the newer version into my go.mod
.
In other words, if I add this line to go.mod
:
replace github.com/Gbps/mymodule => /path/to/mymodule/
Then go mod tidy
works as expected. If I have use /path/to/mymodule/
in my go.work
, it should accomplish the same thing. The benefit being I don't have to taint go.mod
with my local paths, as that is going to need to be committed to remote whereas go.work
isn't.
This is a very big issue for me in the case of creating Docker containers :( I cant use go mod tidy
and copy the dependency graph to the Dockerfile. So every time I build a container I have to download downstream dependencies instead of doing go mod download
.
Golang is considered the language for microservices and In the current state of the go work
it's very hard to work with them in monorepo. Creating multiple-container environment is incredibly hard :(. Basically, every time I hit docker-build
it has to download downstream dependencies. Currently, I have two ways to fix this:
replace
and require
create too many relative dependencies and it's very hard to maintain.I expect 4 things from the golang workspaces
:
go mod
to know about other modules and don't try to download the local package from the internet. Maybe traverse up in the folder hierarchy and check if go.work
exists?go work tidy
and go work download
, because now I have to enter every package and type go mod
commands. I just want to tidy and download everything from one place, and because we already have a path to the modules in go.work
file - it should be trivial.turborepo
, lerna
, and other monorepo management tools.So now, the only thing why I'm using go work
is to have a better codding experience in VsCode. Most of the negative comments that I hear from colleagues about golang are about poor modules/workspaces management. I think, go work
is a very important feature to have and it's a very important direction to go.
I've faced this issue yesterday but my case is a bit different.. My replace
statements are in the go.work
file, which seemed to solve go run
errors, but when attempting go mod tidy
on any of the modules, it is indeed ignored.
The solution is simply to move the replace
statements to the go.mod
files as explained by Gbps, but keeping track of what package uses what internal package is a bit of a hassle. Is this considered a bug also?
go mod tidy
is intended to update the module's self-contained dependencies. It doesn't use the workspace because in general one may work on multiple independently-maintained modules in the same workspace, and if you're preparing an upstream commit you definitely don't want that commit to rely on unpublished local modifications.
go work sync
is intended to update the modules within a workspace to reflect the dependencies selected in that workspace. That is probably what you want if you are working on a set of modules that are all maintained as a single unit.
Folks who are commenting here (and please bear in mind https://go.dev/wiki/NoPlusOne!) — have you considered go work sync
for your use-case? If so, what does go work sync
do (or not do) that makes it unsuitable for your use-case?
@bcmills I'm not sure what the intended workflow is, but I think the feedback here is that the DX is really not intuitive.
Folks who are commenting here (and please bear in mind https://go.dev/wiki/NoPlusOne!) — have you considered go work sync for your use-case? If so, what does go work sync do (or not do) that makes it unsuitable for your use-case?
I wish there were go work tidy
or something similar that would do the same as go mod tidy
for each module in the workspace. Or maybe go work sync
could have a flag for that, or even do it by default. This way one could easily manage a single repository with many modules, keeping everything in sync and tidy :) without having to switch to every single module directory to keep it tidy.
There's an additional issue (I think) I reproduced by cloning this repo:
https://github.com/xmlking/go-workspace
If you try to execute:
go mod verify
It will complain with:
github.com/xmlking/go-workspace/cmd/app1 : missing ziphash: open hash: no such file or directory
github.com/xmlking/go-workspace/cmd/app2 : missing ziphash: open hash: no such file or directory
github.com/xmlking/go-workspace/cmd/app3 : missing ziphash: open hash: no such file or directory
github.com/xmlking/go-workspace/lib : missing ziphash: open hash: no such file or directory
There's something we can do to fix this?
@lopezator, that looks like a separate issue — this one is specifically about go mod tidy
.
Could you file it separately (https://go.dev/issue/new)? Thanks.
go work sync
does not work for me in that scenario either unfortunately. I have a go.work
for a monorepo containing yet to be published packages (the domain also does not exist yet). e.g.:
use (
./foo
./bar
./fubar
)
replace not-yet-created.com/foo/foo v0.0.0-00010101000000-000000000000 => ./foo
However that replace directive seems to be ignored by go work sync
$ go work sync
go: not-yet-created.com/foo/foo@v0.0.0-00010101000000-000000000000: unrecognized import path "not-yet-created.com/foo/foo": https fetch: Get "https://not-yet-created/foo/foo?go-get=1": dial tcp: lookup not-yet-created.com on 127.0.0.53:53: server misbehaving
I've found that the problem with go mod tidy
only happens when a sub-module does not exist in a package.
I've created a simple example to demonstrate the issue: https://github.com/skelouse/go-work-test
Branch master, go run .
and go mod tidy
work as expected
Branch success, go run .
and go mod tidy
both work as expected.
Branch failing, go run .
works, but go mod tidy
fails with:
github.com/skelouse/go-work-test/main imports
github.com/skelouse/go-work-test/foo/newbar: module github.com/skelouse/go-work-test/foo@latest found (v0.0.0-20220809210529-8026891d438c), but does not contain package github.com/skelouse/go-work-test/foo/newbar
I've also created a CLI which uses go.work
in combination with vscode workspaces. Currently have to revert back to hard replaces for certain changes.
Timed out in state WaitingForInfo. Closing.
(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)
The documentation for workspaces
go.work can be used instead of adding replace directives to work across multiple modules.
seems misleading here if not all of the go
commands work with using the go.work
as an override for go.mod
.
I fully understand and for the most part agree that go mod tidy
may not be the solution for addressing tidy
-operations in a go workspace
tree; a go mod tidy
may modify a go.sum
with unpublished references that when synced to a remote/upstream may include references that cannot be reached.
This could be a case of cleaning up/amending the documentation with further clarifications on what use-cases exactly a go workspace
can solve.
I would like to very humbly suggest that we need a go.work.sum
(or similar) file separate from go.sum
, as we are dealing with two separate graphs of modules.
@tliron Coincidentally something (not sure what tool exactly - it was probably done under the hood from the VS Code Go extension) generated a go.work.sum
file in some of my go workspaces.
I had unexpected behavior from go get
related to go work. I'm concurrently developing a library in a directory and an application in a sibling directory. I did an initial commit on the library on main with just a go.mod file, pushed it to a remote (a private bitbucket instance), then started a develop branch. In the application I created a go.work that looked like:
go 1.19
use (
.
../library
)
I am also using vendoring in the application as part of a Docker build process. Everything was working pretty smoothly though, whenever I made a change to the library, I'd commit it, then in the application run:
go get the-module-name/library@hash-of-latest-commit
go mod tidy
go mod vendor
then everything would be up to date. This would work regardless of whether I'd pushed the latest library commit to the remote.
At some point during development, I made a first pull request from the library's develop branch to main on the remote. Once I had done this, if I then tried to go get
the library with a local only commit it errors with an invalid commit hash. I now must push the library commits to the remote in order to update them in the application's go.mod or vendor directory. This makes the go.work file irrelevant.
@tliron, there is already a go.work.sum
file that is populated if needed. (In many cases it is not needed — often the individual go.sum
files in the workspace are sufficient to cover all of the relevant modules.) But that doesn't really seem relevant to this particular issue.
@nine9ths, the behavior of go get
for an unpublished commit is off-topic for this issue. (That's more closely related to #28835.)
@bcmills I did not know this and do not see this file in our environment. Could you point me to the documentation where go.work.sum
is discussed? When is it populated? I assume that if a go.work.sum
exists, then it would be used instead of go.sum
for all operations?
Specifically to our topic, when I run go mod tidy
does it update both go.sum
and go.work.sum
at the same time? Would there be a way to update only go.work.sum
? That last ability would solve all our problems, which is why I raised the topic. During work, go mod tidy
would fail and we know it would fail because ... we are in the midst of working on it. :)
We're having a lot of trouble with this go.work
feature, like others here.
@tliron, documentation for go.work.sum
is #51941. Again, please keep discussion on the issue tracker specific to the issue.
Thought I found a workaround, but it's broken in its own unique ways see here
Turns out go mod tidy
will happily skip attempting to download modules which don't look like URLs - which seemed to have some potential for creative misuse...
So you can use local module names like my-repo/my-module
instead of github.com/my-repo/my-module
just fine...
go mod tidy
behaves itself and doesn't try to download remote versions of local workspace modules
go run
/ go build
work properly
go work sync
works properly...
until someone else wants to import those modules (or any modules in your monorepo which depend upon them)
Seems like all the building blocks are there to support what's discussed in this issue, but there's logic in various tooling which prevents it. (eg. if go get
didn't care if your internal module names matched how they're presented to the world via the go.work
file it would be fine.)
Perhaps worth considering either:
go.work
so that the go.mod
files remain intact and behave as expected@bcmills go work sync only works if you actually have a go.work file. Otherwise it errors.
I have a scripted install process that involves calling go mod tidy followed by a number of go install calls, all part of a continuous integration process.
I was using go.work to be able to test the code in a working branch. However, go mod tidy attempts to grab the code from a release on a main branch. Normally, I want that, but when testing new code in a working branch, I don't.
Trying to use go work sync would mean I would have to change the install scripts to do a go work sync if a go.work file was available, and go mod tidy if not. Not too difficult, but a pain.
However, running go work sync on a new install gives me errors like this:
require github.com/goradd/goradd: version "latest" invalid: must be of the form v1.2.3
So go work sync will not do what is expected, which is to get the latest versions of the packages referred to, except if there is a use directive, in which case it should just use whatever is pointed to. This means I am stuck. I can't call go mod tidy because it will error on a missing package since it will not use the go.work file to find it and so produces an error. And I can't call go work sync because it will not load the latest packages when the go.mod file is asking for it, and so produces an error. Perhaps I am missing an argument that will enable updating of the go.mod file? go help work sync and go work help did not reveal anything helpful in this regard.
IIUC, go mod tidy
downloads external modules declared in import statements and adds them to go.mod
, but go work sync
doesn't traverse source code? That's a fairly major tooling regression for workspace users. Are there other differences?
Independent of the original intent of go mod tidy
and go work sync
, it appears that the go toolchain disagrees with itself here. Either you are using workspaces or you're are not, or is there some inbetween state? When you type go mod tidy
from muscle memory -- I'd assume the intent is "clean up in project the way it's organized" rather than "disregard my project structure and clean up as a standalone module".
As for motivation, many people strongly prefer monorepos for the overwhelming benefits they provide compared to the inconveniences and pitfalls of tightly coupled multi-repo projects. I'm sure Googlers can relate to this :) That said, I can't seem to find a definitive answer on whether monorepos is a first-class use case, and I've read the official docs, tutorials and design doc.
Its always amusing to discover how bad the go monorepo support is 😉. Frankly this is ridiculous - this thread is more than a year old, and instead of simply applying a fix the go maintainers attitude is to ignore it. Which is pretty much buisness as usual regarding anything DX related (see code coverage related issues etc.).
IIUC,
go mod tidy
downloads external modules declared in import statements and adds them togo.mod
, butgo work sync
doesn't traverse source code? That's a fairly major tooling regression for workspace users. Are there other differences?Independent of the original intent of
go mod tidy
andgo work sync
, it appears that the go toolchain disagrees with itself here. Either you are using workspaces or you're are not, or is there some inbetween state? When you typego mod tidy
from muscle memory -- I'd assume the intent is "clean up in project the way it's organized" rather than "disregard my project structure and clean up as a standalone module".As for motivation, many people strongly prefer monorepos for the overwhelming benefits they provide compared to the inconveniences and pitfalls of tightly coupled multi-repo projects. I'm sure Googlers can relate to this :) That said, I can't seem to find a definitive answer on whether monorepos is a first-class use case, and I've read the official docs, tutorials and design doc.
A very commonsense thing to write. As are so many other posts in this thread.
@Goldziher, simply quoting a previous comment and complaining about the lack of activity does not help to advance this conversation (see https://go.dev/wiki/NoPlusOne).
As noted in https://github.com/golang/go/issues/50750#issuecomment-1194322256, we haven't taken action on this issue because we don't have a clear net-positive change to make. If you want to move this conversation forward, please provide constructive feedback on your concrete use-case. Thanks.
@bcmills here is an example: https://github.com/BorzdeG/samples.go-and-flutter-grpc
If we execute go mod tidy
in the backend
directory, we will get an error importing the library that is in public-api/go
and connected via go.work
.
@bcmills Maybe the difficulty to resolve this is related to the nature of go.mod, and go.work things.
How about just add link to the go.work from go.mod to override the go.work's use
or replace
directives first before handling go.mod.
So, we can ensure that, in case of we cannot find go.work from the link (relative path maybe), then just follow the version and URL, but when we have go.work, then we can follow the path of go.work.
Actually, I'm not sure what is the purpose go workspace except seeing monorepo golang code at the same vscode workspace when this cannot be supported.
Maybe add an go work tidy
command similar to go mod tidy
, but understanding go.work
?
@BorzdeG I think go work tidy
is also not a complete solution because in that case, the module should be tightly coupled with go.work. If we can just point out go.work from go.mod, the go module can be still managed independently, but just reference the go.work to get proper library only.
@bcmills Hi, are there any discussion channels for this issue?
From this thread, it seems that this is the intended functionality. This makes workspaces so much less useful than they could be, any updates on discussions, is this being mentioned in some proposals?
I wonder if I am missing something here, but I intended to use workspaces as a more cleaner "replace" alternative when working on a module that imports a package from my own unpublished fork.
E.g. I am working on a github.com/user/project1
and it imports github.com/user/project2
.
The project2
is not published yet and it is in my go.work file imported by a local path.
But then if I need to import new public dependencies in project1
I need to run go mod tidy
to refresh the go.sum and process relative imports, and this fails, since go mod tidy
tries to find project2
.
I wonder if I am missing something here, but I intended to use workspaces as a more cleaner "replace" alternative when working on a module that imports a package from my own unpublished fork.
E.g. I am working on a
github.com/user/project1
and it importsgithub.com/user/project2
.The
project2
is not published yet and it is in my go.work file imported by a local path.But then if I need to import new public dependencies in
project1
I need to rungo mod tidy
to refresh the go.sum and process relative imports, and this fails, sincego mod tidy
tries to findproject2
.
This is exactly how I was expecting it to work, and I have the same issue. I've ended up just using standard go.mod files with replace directives and stripping them out with sed when building my docker images. Workspaces sounded like it would be ideal for this scenario, but sadly not.
Since the go team doesn't seem interested in addressing this issue, perhaps a tool that configures a local proxy using the go.work file could work?
It is very strange that the default assumption in a module that is nested in a workspace wants to depend on pinned published versions of other modules in the same workspace, especially when the language itself was developed in a monorepo to avoid this state of affairs. Gazelle also avoids this state of affairs, but incompletely because it uses go.mod files which go mod tidy cannot correctly manage. I can only assume there is a private tool or fork of the go command used in place of the public go mod tidy
command that has removed any motivation to upstream a fix.
The messaging from users has consistently been that this behavior doesn't actually serve any use cases that could not still be served by simple checks for workspace context. Even if you want to publish derivative single module repositories in public mirror repos, nothing about changing go mod behavior within the context of a go workspace would break that use case.
Since the go team doesn't seem interested in addressing this issue, perhaps a tool that configures a local proxy using the go.work file could work?
As you devs work out how to fix this, please keep this use case in mind. One of the reasons to use a workspace is the scenario where we want to build multiple binaries, and those binaries share some common code or other resources. A client-server pair of binaries using a common protobuf makes a good example. Here one might use a layout like this:
myapp
go.work
bin
client.exe
server.exe
cmd
client
client.go
go.mod
server
server.go
go.mod
proto
myapp.proto
pkg
common1
common1.go
common2
common2.go
The desire to use a workspace here is in order to work on these two binaries as a single unit. This is because:
When working under this use case, the naive go newcomer may expect that running go mod tidy at the top of the workspace (as well as some other commands) should produce the same result as if they ran it separately inside lower directories. This is because they are thinking in the mindset that these are two separate binaries. Granted that's not how the existing tool options are supposed to work today, but in the long run it would be nice to have options in the tools that support using a workspace for this use case, to maintain what are essentially independent binaries that just happen to need to be maintained together as a single unit.
Whatever the answer ends up being, it would be good to explain the best way to manage this type of scenario somewhere in the workspace docs as well.
Depending on what a user's expectations are, this may have some overlap with https://github.com/golang/go/issues/40302
@mdepot What's the use case for the client and server being in different modules from each other?
@mdepot What's the use case for the client and server being in different modules from each other?
@matloob can we not XY-problem this and take as given that it is a valid use case to have a client and a server be in different modules, especially considering it is commonplace for them to be in entirely different repos?
Otherwise this turns into a tangent of "why are monorepos, especially ones where packages within can have dependencies on other packages' source, beneficial", which is deeper topic than needs to be explored here, and it can just as easily be taken a given that a non-trivial number of organizations (and therefore go toolchain users) have decided this has some good answers.
For further clarification, I used the term "package" above in the general sense, not the golang-specific sense; pretend I said "module" if you prefer.
@matloob: i'm in the same situation as @mdepot. In many cases, the client is free / open-source while the server is proprietary / closed-source. There are also cases where one side has far more dependencies than the other (i.e. why force people to pull a fat Electron client when they only want a server side that is strictly files & sockets?).
Whatever the answer ends up being, it would be good to explain the best way to manage this type of scenario somewhere in the workspace docs as well.
We also use a similar structure for the above mentioned reason. When running Go tidy from the root, it took us by surprise that it ignored the go.work.
A quick fix was running:
for f in $(find . -name go.mod)
do (cd $(dirname $f); go mod tidy)
done
But that is just a temporary solution, for a missing feature.
@dkrieger, generally we expect “dependencies on other packages' source” to mean something that can be fetched as a module, from a (public or private) GOPROXY or version-control host. It isn't obvious why one would have dependencies that can't at least be redirected for go get
using an insteadOf
directive in the user's .gitconfig
.
What version of Go are you using (
go version
)?What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
Minimal reproducing repository: https://github.com/bozaro/go-work-play/tree/go-mod-tidy
Full script:
What did you expect to see?
I expect
go.mod
andgo.sum
updated with current working copy state.What did you see instead?
I see that
go mod tidy
try to get modules forshared
go modules from repository ignoringgo.work
content.