bozaro commented 2 years ago

What version of Go are you using (`go version`)?

$ go version
go version go1.18beta1 linux/amd64

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/bozaro/.cache/go-build"
GOENV="/home/bozaro/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/bozaro/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/bozaro/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.18"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.18/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18beta1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/bozaro/github/go-work-play/tools/go.mod"
GOWORK="/home/bozaro/github/go-work-play/go.work"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4292218461=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Minimal reproducing repository: https://github.com/bozaro/go-work-play/tree/go-mod-tidy

Full script:

$ git clone https://github.com/bozaro/go-work-play.git -b go-mod-tidy
$ cd go-work-play/tools
$ go mod tidy
go: finding module for package github.com/bozaro/go/go-work-play/shared/foo
github.com/bozaro/go/go-work-play/tools/hello imports
    github.com/bozaro/go/go-work-play/shared/foo: module github.com/bozaro/go@latest found (v0.0.0-20200925035954-2333c6299f34), but does not contain package github.com/bozaro/go/go-work-play/shared/foo

What did you expect to see?

I expect go.mod and go.sum updated with current working copy state.

What did you see instead?

I see that go mod tidy try to get modules for shared go modules from repository ignoring go.work content.

bcmills commented 8 months ago

@BourgeoisBear

There are also cases where one side has far more dependencies than the other (i.e. why force people to pull a fat Electron client when they only want a server side that is strictly files & sockets?).

That was the motivation for adding module graph pruning in Go 1.17. Are there cases of this sort where graph pruning is ineffective? If so, can you give more detail about the concrete problems (ideally as a separate issue)?

dkrieger commented 8 months ago

@dkrieger, generally we expect “dependencies on other packages' source” to mean something that can be fetched as a module, from a (public or private) GOPROXY or version-control host. It isn't obvious why one would have dependencies that can't at least be redirected for go get using an insteadOf directive in the user's .gitconfig.

@bcmills The obvious case is when you have a monorepo with multiple go modules, orchestrated with that other Google tool bazel/blaze. Being able to publish updates to two or more modules in your monorepo in the same commit is high on the list of benefits of a monorepo.

bcmills commented 8 months ago

@dkrieger, can you give some more detail about why the monorepo is structured that way? (We're trying to better understand the underlying use-cases so that we can address them holistically.)

We're familiar with https://github.com/googleapis/google-cloud-go, which IIUC uses multiple modules so that they can tag its various APIs at different levels of stability (for example, have some APIs stabilized at v1 while others are still at v0). But the discussion on this issue has been mostly focused on dependencies that don't have any versions published at all, which would imply that that's not the same underlying motivation.

BourgeoisBear commented 8 months ago

@bcmills, "pull" as in git/hg of a repo with large assets (like a GUI with many rasters/videos) when I don't need it. No complaints on compile time or binary sizes.

bcmills commented 8 months ago

Hmm, here's a thought. For #50603, we're going to need the ability to inspect the local repo containing a given repository to determine its version information.

In theory, go work sync could do the same thing for every main module in the workspace, so that if some module in the workspace depends on it that dependency could be updated to refer to the version actually found in the workspace.

But there is a bit of a chicken-and-egg problem: the version of a checked-out repo in the workspace depends on its commit hash, which depends on the contents of its files. If there is a cycle in the requirement graph (which is normally allowed), then go work sync would become unstable: each time it writes out a go.mod file in one module, the versions it records in the other go.mod files would be updated to the new hash, causing an additional go.mod write. The only ways to reach a fixed point are:

Decouple the actual versions written to the go.mod files from the versions found in the workspace, such as by resolving upstream commits.
Have go work sync intentionally remove references to the workspace modules from the go.mod files, presumably to be filled in later.
Have the user explicitly specify versions to assign to the workspace modules, instead of trying to infer them from the repo.

dkrieger commented 8 months ago

@bcmills let me start with a language agnostic description of why we would structure a monorepo in this way, then I'm happy to field any go-specific follow-up questions you might have.

At our company (and I'd suggest most companies), most of our code is not public. Our deployable artifacts are mostly containers that we deploy to kubernetes.

For some given deployable slice of the codebase, we may deploy it as one or more services, and a given chunk of code may appear in one or more containers -- in a common scenario where it appears in multiple containers, we may have a fatter container that aggregates multiple domains (let's say each domain is represented by one grpc service, and we pack those into single networked service, i.e. many:1 domain:service) and a smaller container that is a microservice (i.e. 1:1 domain:service). This enables sophisticated traffic shaping when both are deployed, but in more simpler scenarios it means we can periodically move predictable-traffic and/or high-reliability domains into the aggregate service, and move bursty and/or low-reliability domains into their own microservices -- basically, we preserve domain boundaries and the flexibility of deploying as a microservice or in an aggregate/monolith as we see fit, balancing performance, complexity, efficiency, and reliability over time.

At lower levels, we may have various utility libraries that get used by many deployable slices of the codebase.

Now, for all that private code, we don't really care about publishing these for consumption from other repos, whether inside or outside the organization. As such, we don't have to worry about versioning in the traditional sense -- any commit on the trunk branch (or, by convention, any merge commit) can be built and deployed, and the commit hash is the "version" for everything. Because everything we do is in the monorepo, we not only know every consumer for X service, but they share the same build and IDE context. We can make breaking changes to a service and update every consumer in a single commit if we want to (in practice we'd make a backwards compatible change followed by cleaning up the deprecated API when talking about services, but in the case of libraries we can do it all in one shot).

Still, we want strong boundaries between different parts of our codebase, and the universally strongest boundary is to structure it as the unit a given language uses for external dependencies -- in node, this is a package; in go, this is a module. We don't actually want to put package registries in the middle of this though.

When I look at my string util package ("module" in go), I want to be able to look at the go.mod file and see just the dependencies for that, not polluted with a single line that has to do with some consumer of the string util go module, or some other completely unrelated go module. If it doesn't depend on any other go modules in the monorepo, it could be built with just the standard go toolchain.

For higher level go modules in the monorepo (e.g. a networked service), if I don't want them published for consumption outside the monorepo, they will have dependencies on external libs and some other internal modules in the monorepo. I'd like to be able to go mod tidy --workspace and not have it barf, behaving how it currently does for external libs, but for other modules in the workspace (as defined in the go.work file), it will point to those local sources. This allows for making logical changes to module apis in a single commit, updating consumers that would otherwise break, without going through the toil of publishing each update in the dependency graph first. With bazel, every part of thus workflow works except for the go mod tidy portion, because go mod tidy doesn't understand how to handle this.

matloob commented 8 months ago

@dkrieger Could you speak about how you use Bazel together with modules? It was my understanding that Bazel didn't support modules. Is that incorrect? Why do you need modules if you're using Bazel to do your builds?

Do you have a Bazel WORKSPACE for each of your modules? My understanding is that the workspace in Bazel is what maps closest to a Go module.

dkrieger commented 8 months ago

@matloob Bazel with rules_go and gazelle lets you configure gazelle to source your workspace-level go dependencies from the union of every go.mod in your bazel workspace. A bazel workspace maps to a go.work, if you are using go modules and workspace. The only pain point is the missing ability to have go mod tidy manage a given go.mod for you that uses your unpublished/unversioned sources for other modules in the same go/bazel workspace

matloob commented 8 months ago

@dkrieger I didn't realize that Bazel supported that.

I want to try to step back a bit. The reason I'm so hesitant about supporting this is that we don't want the tooling to encourage folks to use multiple modules when they can work with a single one. The motivation that I understand for having a multi module monorepo is that the modules are distributed separately from each other. That is: the modules appear in a monorepo, but they are also dependencies of other external modules. For those cases go mod tidy should work fine because each module is able to stand on its own as a dependency of an external project.

So I want to understand your use case better. I see that you mentioned the string util libraries. Are those depended on by other modules outside of your monorepo? Are they built separately from the rest of your monorepo?

dkrieger commented 8 months ago

@matloob

The reason I'm so hesitant about supporting this is that we don't want the tooling to encourage folks to use multiple modules when they can work with a single one.

I have a couple responses to this:

an optional cli flag does not encourage its usage, and the docs can express (non-)recommended use cases. In a pnpm monorepo, only when you add the --workspace flag when adding a dependency will it prefer the local workspace sources (vs some pinned version available on the registry) when adding an intra-workspace dep.
I think borrowing the python mantra of "we're all adults" is appropriate here. Ergonomics and defaults are the appropriate ways to signal recommended patterns, not capabilities. In python, prefixing with an underscore signals "this isn't meant for external use", but you can still do it if you feel you have a good reason to; this works out really well in practice

The motivation that I understand for having a multi module monorepo is that the modules are distributed separately from each other.

For all intents and purposes, I'd say that the use case I'm describing does involve distributing modules separately from one another; whether they're distributed in the conventional go way or not, whether they're published to a public registry (including public github repo) or not -- these are downstream decisions. In all cases, they're possibly (and in practice, often) developed in lock step, and contracts can be changed in backwards incompatible ways in a single atomic commit. Again, the main effect is eliminating the toil of traversing your dependency graph and the logical change you're wanting to make becoming distributed over the dimension of git-commit-time

The result is much cleaner than having a huge module with packages that have no logical relationship with one another. Every module I create tends to consist of several packages, and those packages all pertain to the logic of that module. The 2nd level of ordering that modules provide is incredibly useful, as it (1) makes it easier to understand the transitive closure of dependencies of any piece of code without noise and (2) makes it easier to understand the interrelated packages that exist in that module. This is just as true whether or not I publish my modules in a way that they can be retrieved via go get in some external repo/organization.

In practice, I'd only invest in versioning/publishing for consumption in external repos if I want legacy pre-monorepo code to use them. I don't want to extract a network of packages from a monolithic module when that determination is made, I want to make small changes to my distribution logic (and introduce versioning)

nemith commented 8 months ago

That was the motivation for adding module graph pruning in Go 1.17. Are there cases of this sort where graph pruning is ineffective? If so, can you give more detail about the concrete problems (ideally as a separate issue)?

When downloading a module the entire repo is checked out. So if you have a single go.mod for a monorepo you easily run into an issue where you can hit the limits of size of repo (I can't remember what the limits are but it's hardcoded into the go command). Using submodules does inform git to ignore directories.

I am specifically running into this problem with trying to add new modules to an existing repo. New module is created, needs to be referenced by other modules in the monorepo but go get and go mod tidy don't work as the module doesn't exist upstream yet. This is breaking a normal workflow for coding and means we have to commit a dummy mod first (and get code review on it) before hacking further.

Replace works here, but sorta defeats the purpose of go.work.

spekary commented 8 months ago

means we have to commit a dummy mod first (and get code review on it) before hacking further

Yep, that is how I had to work around it. I don't think it has anything to do with limits on the size of a repo though.

nemith commented 8 months ago

means we have to commit a dummy mod first (and get code review on it) before hacking further

Yep, that is how I had to work around it. I don't think it has anything to do with limits on the size of a repo though.

What i mean is that I am using (or rather want to us) multiple modules to get around the size of my monorepo vs just having a single go.mod for the entire repo.

There are two problems with large monorepos and a single go.mod when it comes to size. One is just download size, it takes a long time for consumers of one project in the the monorepo to download and cache the entire thing. The other is hard limits.

For all download type it looks like it defaults to 10<<20 (or around 524MB) which is a lot for a repo, but we use git-lfs which is resolved here and includes any binaries you may want to stuff into your monorepo:

https://github.com/golang/go/blob/master/src/cmd/go/internal/modfetch/codehost/codehost.go#L35 https://github.com/golang/go/blob/master/src/cmd/go/internal/modfetch/proxy.go#L435 https://github.com/golang/go/blob/master/src/cmd/go/internal/modfetch/codehost/vcs.go#L465 https://github.com/golang/go/blob/master/src/cmd/go/internal/modfetch/coderepo.go#L1087

Submodules is a great way to solve this, but because of resolving versions and the chicken-and-egg problem they are hard to work with and hard for my coworkers to stomach.

dkrieger commented 8 months ago

An important underlying theme here is that monorepos are not monolithic codebases, nor are they distributed eventually consistent codebases where you necessarily version and publish each part to some registry. They are no less modular than multi-repo/polyrepo -- to the contrary, they promote mindful modularity by reducing the incidental disincentives to modular design. That they don't encourage/require long-lived support branches in the form of versioning in many circumstances is frankly unrelated to modularity.

If we can agree that this is at least a defensible position, let's try to shift the focus to how we can decouple go module tooling behavior from opinions for/against monorepo adoption without breaking userland.

rsc commented 8 months ago

"go mod" commands are scoped to modules when making changes; there is no workspace in go mod sub-commands. This is why "go work vendor" is different from "go mod vendor". So "go mod tidy" should not know about go.work any more than the other commands. (Some of the read-only commands like "go mod graph" are workspace-aware, but that's less problematic than read-write commands.)

It seems like "go work sync" is what we should be focusing on. What does it do or not do that is inappropriate?

brianbraunstein commented 8 months ago

@rsc Hi Ross, thanks for looking into this.

What does it do or not do that is inappropriate?

Scenario

Imagine this scenario, you're planning on creating a repo with the following:

root
├── lib
│   ├── some_code.go  // package example.com/lib
├── prog
│   └── main.go   // package example.com/prog

prog/main.go uses a function from lib/some_code.go.
prog/main.go also uses a function from thirdparty.com/foolib
You want to make lib and prog as independent modules for various reasons (I think these are also mentioned by others above):
- lib will be useful in other places than just prog, and in those other places, pulling in prog isn't appropriate.
- Limiting scope to keep things organize and limit cognitive load when focusing on a single module.

Problem

So now you learn about "go workspaces" that seem like they could help with this and you want to quickly try it out. You make the above file tree, then do:

cd root
go work init
go work use -r .

cd root/lib
go mod init
go mod tidy

cd root/prog
go mod init

# so far so good

go mod tidy
# ERROR: ...cannot find module providing package example.com/lib...

To workaround you can do:

cd root/prog

go mod edit -replace=example.com/lib ../lib
go mod tidy
go mod edit -dropreplace=example.com/lib

go build

But that's obviously silly because you might as well not use workspaces at that point.

Possible Solution 1: `go work modtidy`

Make something like go work modtidy that does the job of go mod tidy, but like go build, is aware of the go.work file.

Possible Solution 2: Clarify the purpose of workspaces

One solution might be to just add clarification in the documentation about the intended use cases for workspaces, and more importantly, what use cases are NOT appropriate for workspaces.

https://go.dev/doc/tutorial/workspaces says:

With multi-module workspaces, you can tell the Go command that you’re writing code in multiple modules at the same time and easily build and run code in those modules"

I think this is leading people astray. In their mind, they read this and think of the scenario I've given here. One could argue that in this scenario you should not use workspace and instead either:

just check root into github/etc so that all dependency modules, even the local ones, are always accessible for go mod tidy to actually work.
just use replace in this case.

Tutorial use case maybe shouldn't use workspaces too...

Adding some clarification would be helpful for me because so far I'm failing to see the compelling use case for workspaces. In the example given in https://go.dev/doc/tutorial/workspaces, it seems to me that it would have been better done with replace. At the very end of the tutorial it says:

...Now, to properly release these modules we’d need to make a release of the golang.org/x/example/hello module...Once the release is done, we can increase the requirement on the golang.org/x/example/hello module in hello/go.mod

With replace, if you accidentally forget to submit the change to the dependency first and send the code review out, your reviewer will say "hey, what's with this replace business!? are you running your code against a locally tweaked version of that library!? please don't check this in, it will break for everyone else". Also, if the dependent does accidentally get submitted first and breaks everyone else, the error message will make it will be pretty clear what happened (replace pointing to some weird ../../foo location). Using workspaces like the tutorial, the reviewer is much less likely to notice, there's no trace of what happened in the code review, and when the breakage happens it will be more confusing to root cause, especially if it's a behavior change to an existing function that wouldn't be caught at compile time.

So maybe it would be helpful to have a different use case in the tutorial that really shows where workspaces shine.

dkrieger commented 8 months ago

@brianbraunstein

Possible Solution 2: Clarify the purpose of workspaces

Respectfully, this is not a solution, it's a temporary mitigation at best. Not supporting atomic updates spanning multiple workspace modules defies the purpose of a workspace in the general sense of the term, as it is observed across languages and monorepo build tools. Whether we get go mod tidy --workspace, go work modtidy, or some other cli for this use case has no bearing on whether it is a sensible use case to support or not. If we can agree it satisfies a standard of "plausibly sensible", that should be enough to move forward, at which point the finer details of the appropriate cli API can be worked out.

Updating go.mod files appears to be within the purview of go mod. Workspace management is sometimes within the purview of go work, but not exclusively, and in this case, its not really workspace management, but rather module management that could be extended to be workspace-aware. Unless there is another issue with more discussion on this topic (I have not found one), advancing the discussion here whether or not it ends up being a go mod subcommand in the end seems appropriate. If there's buy in for support, then, despite my own misgivings that go work sync is the appropriate domain, that should stay on the table, but gating further consideration by asking this discussion to be reframed as a go work sync deficiency is not constructive IMO @rsc .

atdiar commented 8 months ago

I don't know whether it is related but one issue that I have encountered is, while building something similar to gonew, when downloading a module in a workspace, the go.work file needed to be updated manually.

Perhaps that go work sync could also work bottom up (via a flag?) to add this module/directory to the module list in the go.work file?

If that makes sense.

dpifke commented 8 months ago

If a different command should be substituted for go mod tidy when using workspaces, I think we should update the various places where running other commands instruct the user to run it:

https://cs.opensource.google/go/go/+/refs/tags/go1.22.0:src/cmd/go/internal/modget/get.go;l=1781 https://cs.opensource.google/go/go/+/refs/tags/go1.22.0:src/cmd/go/internal/modload/help.go;l=53 https://cs.opensource.google/go/go/+/refs/tags/go1.22.0:src/cmd/go/internal/modload/init.go;l=648 https://cs.opensource.google/go/go/+/refs/tags/go1.22.0:src/cmd/go/internal/modload/init.go;l=651 https://cs.opensource.google/go/go/+/refs/tags/go1.22.0:src/cmd/go/internal/modload/load.go;l=2023 https://cs.opensource.google/go/go/+/refs/tags/go1.22.0:src/cmd/go/internal/modload/load.go;l=2029 https://cs.opensource.google/go/go/+/refs/tags/go1.22.0:src/cmd/go/internal/modload/load.go;l=2032

This might also be a good way to audit go mod tidy use cases and make sure they are covered by whatever is the equivalent workspace command.

Being told by the tooling to run go mod tidy, and having that fail, is what brought me to this issue originally. I've since given up on using workspaces and instead use replace directives in go.mod, along with a hacky Git pre-commit hook to filter those out when committing. (I'm probably holding it wrong.)

ohir commented 7 months ago

@matloob

So I want to understand your use case better.

Many use cases were described by participants here. Mine is that I do not want to publish dummy stub modules before I can start to work. I can buy rsc explanation why go mod tidy should not know about go.work. But certainly go work should be able to recursively tidy all use pieces declared in go.work workspace, akin to go work sync. Thus please consider:

go work tidy

that does not try to reach to unpublished modules. Ie. replaced to the => ./local/path

matloob commented 7 months ago

@dpifke Thanks, that's a useful perspective. We don't want our tools confusing users with the instructions they give. I'm interested if solving the error messages is the primary reason most people want 'tidy' for these use cases.

@brianbraunstein I don't think 'tidyness' is the right concept here. I don't think we should call a go.mod file tidy if it can't be used (alone) to build a module. If go work sync doesn't work we'll need to think of some other clean concept we can use to express that though the modules don't work standalone they can be used in a workspace concept.

@ohir I don't know if tidy is the right concept here. Why wouldn't go work sync work for your use case.

@dkrieger I don't think we should frame this discussion around what the concept of a "Workspace" is in other language ecosystems. We want to build what make the most sense for the Go module system.

ohir commented 7 months ago

@matloob

Why wouldn't go work sync work for your use case

It still reaches to the net even if it has everything replaced or internal. It should not.

Repro tree attached (usage: patch -p0 < goworksync_repro.patch)

goworksync_repro.patch

matloob commented 7 months ago

@ohir

When I disable the network your repro case doesn't produce an error when the two replaces in the go.work are there. But if I remove them I do get an error. Is that what you're seeing too? If not, what version of Go are you running?

ohir commented 7 months ago

@matloob

There is no problem with sync if workstation network (local) is down. Problem arises when the workstation network is on but module repo is not exposed – either due to the firewall rules, or because it hasn't been really set up.

Since sync tries to contact fairbe.org domain looking for the module repo it hangs for a longer while (till Dial timeouts). With real tree having dozens of modules it takes more noticeable time.

This is a wider problem: go get also goes astray on a repo with unpublished/unreachable module: It errs with [...] Get "https://fairbe.org/moda?go-get=1": net/http: TLS handshake timeout.

The only working solution for now is to replenish all go.mod files with respective replaces, too. What nullifies all, or almost all, benefits of the workspace mode.

go version go1.22.1 darwin/amd64
GOPRIVATE=fairbe.org,example.com
GONOPROXY=fairbe.org,example.com
GONOSUMDB=fairbe.org,example.com

Of course fairbe.org proxy is not reachable from the outside. But even if that were the case, I don't see why the tool reaches for the repo if it has been told that the replacement is on the local drive.

Summary: Workspace can not be reliably set up for a project that is even partially isolated, ie. it contains an unpublished/unreachable module(s). For the go work sync and go get to work seamlessly everything in the project, including isolated parts, MUST be reachable over the network - or be replaced directly in impoter's go.mod file.

a-pav commented 7 months ago

Summary: Workspace can not be reliably set up for a project that is even partially isolated, ie. it contains an unpublished/unreachable module(s). For the go work sync and go get to work seamlessly everything in the project, including isolated parts, MUST be reachable over the network - or be replaced directly in impoter's go.mod file.

I don't understand why this discussion has continued for this long, honestly, and yet there's no progress to be seen. Your summary seems to sum it up perfectly and I think it's the reason I subscribed to this issue a long time ago: My expectation was to use go.work for the scenario that I want to use an unpublished/locally modified module, but I wasn't able to that. Seems simple enough!

ohir commented 7 months ago

@matloob FYI: neither you can easily vendor an unpublished module in the workspace.

After adding vendor/btea module to the workspace (unpublished fairbe.org/btea) then trying to import/use it resulted in a few minutes of delay in the IDE (awaiting network timeouts) then gopls throwed at me a full bucket of errors. Interestingly — import errors in mostly unrelated modules. Not even stdlib pieces could be found.

Filtered excerpts follows:

[cmd/smth imports fairbe.org/btea module for btea.FromBELE] : "undefined: FromBELE" <= this should come from vendor/btea, it did not. Until replaced explicit in the cmd/go.mod

[other module - imported by cmd/smth but NOT importing fairbe.org/btea at all

: "could not import golang.org/x/crypto/chacha20 (missing metadata for import of \"golang.org/x/crypto/chacha20\")",
: "fairbe.org/btea@v0.0.1: unrecognized import path \"fairbe.org/btea\": https fetch: Get \"https://fairbe.org/btea?go-get=1\": net/http: TLS handshake timeout",
: "could not import fmt (missing metadata for import of \"fmt\")",
: "could not import crypto/ecdh (missing metadata for import of \"crypto/ecdh\")",
: "could not import crypto (missing metadata for import of \"crypto\")",
: "could not import golang.org/x/crypto/chacha20 (no required module provides package \"golang.org/x/crypto/chacha20\")",

To sort this mess out an explicit replace had to be added to the (cmd's) go.mod replace fairbe.org/btea v0.0.1 => ../../../vendor/btea.

I hear "its a gopls issue". No, it is not just gopls – it is a problem of workspaces still being a patchwork on the side of other tools mostly being unaware of the go.work replace directives (@rsc).

I would like my "Summary" warning to be included in the docs of the workspaces for the time being - to save headaches for others trying to add some isolated module to their workspace. Thanks.

GustavoStingelin commented 4 months ago

Hey everyone,

If you declare the local shared module only with the folder name, you can import it directly in the other module Go files without needing a remote go.mod.

By example:

myproject/
├── go.work
├── api/
│   ├── cmd/
│   │   └── main.go
│   └── go.mod
└── shared/
    ├── go.mod
    └── middleware/
        └── middleware.go

$ cat go.work
    go 1.22.1

    use (
            ./api
            ./shared
    )

$ cat shared/go.mod 
    module shared

    go 1.22.1

    require (
       ...3rdparty/lib
    )

$ cat api/cmd/main.go 
    package main

    import (
            "shared/middleware"
    )

$ cat api/go.mod 
    module api

    go 1.22.1

    require (
       ...3rdparty/lib
    )

dkrieger commented 4 months ago

Hey everyone,

If you declare the local shared module only with the folder name, you can import it directly in the other module Go files without needing a remote go.mod.

By example:

myproject/
├── go.work
├── api/
│   ├── cmd/
│   │   └── main.go
│   └── go.mod
└── shared/
    ├── go.mod
    └── middleware/
        └── middleware.go

$ cat go.work
    go 1.22.1

    use (
            ./api
            ./shared
    )

$ cat shared/go.mod 
    module shared

    go 1.22.1

    require (
       ...3rdparty/lib
    )

$ cat api/cmd/main.go 
    package main

    import (
            "shared/middleware"
    )

$ cat api/go.mod 
    module api

    go 1.22.1

    require (
       ...3rdparty/lib
    )

Now try running go mod tidy in myproject/api

GustavoStingelin commented 4 months ago

@dkrieger all looks good here! No errors or warnings.

nyanpassu commented 4 months ago

Hey everyone,

If you declare the local shared module only with the folder name, you can import it directly in the other module Go files without needing a remote go.mod.

By example:

myproject/
├── go.work
├── api/
│   ├── cmd/
│   │   └── main.go
│   └── go.mod
└── shared/
    ├── go.mod
    └── middleware/
        └── middleware.go

$ cat go.work
    go 1.22.1

    use (
            ./api
            ./shared
    )

$ cat shared/go.mod 
    module shared

    go 1.22.1

    require (
       ...3rdparty/lib
    )

$ cat api/cmd/main.go 
    package main

    import (
            "shared/middleware"
    )

$ cat api/go.mod 
    module api

    go 1.22.1

    require (
       ...3rdparty/lib
    )

it only makes go mod thinks the module is in the std. compiling will fail.

GustavoStingelin commented 1 month ago

Hey everyone,

If you declare the local shared module only with the folder name, you can import it directly in the other module Go files without needing a remote go.mod.

By example:


myproject/

├── go.work

├── api/

│   ├── cmd/

│   │   └── main.go

│   └── go.mod

└── shared/

    ├── go.mod

    └── middleware/

        └── middleware.go

$ cat go.work

    go 1.22.1

    use (

            ./api

            ./shared

    )

$ cat shared/go.mod 

    module shared

    go 1.22.1

    require (

       ...3rdparty/lib

    )

$ cat api/cmd/main.go 

    package main

    import (

            "shared/middleware"

    )

$ cat api/go.mod 

    module api

    go 1.22.1

    require (

       ...3rdparty/lib

    )

it only makes go mod thinks the module is in the std. compiling will fail.

I have this project in production.

rufreakde commented 1 month ago

Since this thread seems to summarize many different opinions and ideas as to what to change for the go.work behaviour let me add my two things I would love to have for "early development" usability.

1 dependency management for "initial setup"

I personally hate it that if I work on several new submodules i would need to release an initial version first for all of them to be able to reference them correctly on a main go.mod file. Otherwise the IDE always complains...

why not allow to reference inside go.mod something like a keyword to emphasize that this is handled by go.work file and is not maintained currently (yet) against a stable version.

something linke:

require https://github.com/google/uuid vWorkspace

linters can also work with such a keyword showing the warning for this... builds could fail initially when keyword is found. Same for PR checks. And for initial setup you can finish a submodule with main module tests before pushing or releasing it.

Currently my workaround is: to push it to a branch and use the branch as a version this could help others as well currently maybe.

With this keyword also go mod tidy would just work. Because the expectation of "this is not a final version" is fulfilled and IDE compiler etc still works without extra effort of manual branch versioning.

2 extra work for submodule tidyness

The second point I read here somewhere in the chat already and it is the need to "cd" into all directories to "go mod tidy" all submodules.

Workaround is to edit the make file:


tidy: ## Tidy go modules
    go work sync #sync root
    cd apis/A/v1alpha1 && go mod tidy # tidy all submodules
    cd apis/B/v1alpha1 && go mod tidy
        ...
        go mod tidy # tidy main module

So here I would love an option to do something like:

go mod tidy --recursive #(and if there is no main module it should still look into subfolders...)

I think the "featureset" that is available is usable but with manual effort which is confusing as most of the things where also possible with manual effort without the go.work file and this spikes confusion.

conclusion:

IMHO this two small changes would come a long way in terms of "Developer happiness"

leemeichin commented 1 month ago

To add my feedback, we've been using go in our monorepo and I think the confusion fundamentally boils down the decision that workspaces are an override for local development, but according to guidelines shouldn't be production, and therefore go mod and go work are orthogonal.

We have a setup like:

./services/...
./packages/go/...

We used to have a go.mod at the top level, but that made every package share the same set of dependencies. Creating a workspace seemed like a nice approach (like yarn workspaces in JS for monorepos), and then we'd set up each package in services and packages/go with their own go.mod file. The result is that we have to micromanage it with replace directives now, and those can only be added by hand or through the CLI.

We also tried deleting the workspace and running go mod tidy again, and it doesn't understand imports from other parts of the same repo, yet the code builds successfully.

From my perspective, I feel like the tooling is unintuitive and is probably the most 'magical' part of the language, considering how methodical and obvious the rest of the language is.

golang / go

cmd/go: `go mod tidy` ignores `go.work` file #50750

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

What did you do?

What did you expect to see?

What did you see instead?

Scenario

Problem

Possible Solution 1: `go work modtidy`

Possible Solution 2: Clarify the purpose of workspaces

Tutorial use case maybe shouldn't use workspaces too...

1 dependency management for "initial setup"

2 extra work for submodule tidyness

conclusion:

golang / go

cmd/go: `go mod tidy` ignores `go.work` file #50750

What version of Go are you using (go version)?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

Scenario

Problem

Possible Solution 1: go work modtidy

Possible Solution 2: Clarify the purpose of workspaces

Tutorial use case maybe shouldn't use workspaces too...

1 dependency management for "initial setup"

2 extra work for submodule tidyness

conclusion:

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

Possible Solution 1: `go work modtidy`