cmd/go: 'go install' should install executables in module mode outside a module

jayconrod commented 4 years ago

Authors: Jay Conrod, Daniel Martí

Last Updated: 2020-09-29

Design doc: CL 243077 Comments on the CL are preferred over comments on this issue.

Abstract

Authors of executables need a simple, reliable, consistent way for users to build and install exectuables in module mode without updating module requirements in the current module's go.mod file.

Background

go get is used to download and install executables, but it's also responsible for managing dependencies in go.mod files. This causes confusion and unintended side effects: for example, the command go get golang.org/x/tools/gopls builds and installs gopls. If there's a go.mod file in the current directory or any parent, this command also adds a requirement on the module golang.org/x/tools/gopls, which is usually not intended. When GO111MODULE is not set, go get will also run in GOPATH mode when invoked outside a module.

These problems lead authors to write complex installation commands such as:

(cd $(mktemp -d); GO111MODULE=on go get golang.org/x/tools/gopls)

Proposal

We propose augmenting the go install command to build and install packages at specific versions, regardless of the current module context.

go install golang.org/x/tools/gopls@v0.4.4

To eliminate redundancy and confusion, we also propose deprecating and removing go get functionality for building and installing packages.

Details

The new go install behavior will be enabled when an argument has a version suffix like @latest or @v1.5.2. Currently, go install does not allow version suffixes. When a version suffix is used:

go install runs in module mode, regardless of whether a go.mod file is present. If GO111MODULE=off, go install reports an error, similar to what go mod download and other module commands do.
go install acts as if no go.mod file is present in the current directory or parent directory.
No module will be considered the "main" module.
Errors are reported in some cases to ensure that consistent versions of dependencies are used by users and module authors. See Rationale below.
- Command line arguments must not be meta-patterns (all, std, cmd) or local directories (./foo, /tmp/bar).
- Command line arguments must refer to main packages (executables). If a argument has a wildcard (...), it will only match main packages.
- Command line arguments must refer to packages in one module at a specific version. All version suffixes must be identical. The versions of the installed packages' dependencies are determined by that module's go.mod file (if it has one).
- If that module has a go.mod file, it must not contain directives that would cause it to be interpreted differently if the module were the main module. In particular, it must not contain replace or exclude directives.

If go install has arguments without version suffixes, its behavior will not change. It will operate in the context of the main module. If run in module mode outside of a module, go install will report an error.

With these restrictions, users can install executables using consistent commands. Authors can provide simple installation instructions without worrying about the user's working directory.

With this change, go install would overlap with go get even more, so we also propose deprecating and removing the ability for go get to install packages.

In Go 1.16, when go get is invoked outside a module or when go get is invoked without the -d flag with arguments matching one or more main packages, go get would print a deprecation warning recommending an equivalent go install command.
In a later release (likely Go 1.17), go get would no longer build or install packages. The -d flag would be enabled by default. Setting -d=false would be an error. If go get is invoked outside a module, it would print an error recommending an equivalent go install command.

Examples

# Install a single executable at the latest version
$ go install example.com/cmd/tool@latest

# Install multiple executables at the latest version
$ go install example.com/cmd/...@latest

# Install at a specific version
$ go install example.com/cmd/tool@v1.4.2

Current `go install` and `go get` functionality

go install is used for building and installing packages within the context of the main module. go install reports an error when invoked outside of a module or when given arguments with version queries like @latest.

go get is used both for updating module dependencies in go.mod and for building and installing executables. go get also works differently depending on whether it's invoked inside or outside of a module.

These overlapping responsibilities lead to confusion. Ideally, we would have one command (go install) for installing executables and one command (go get) for changing dependencies.

Currently, when go get is invoked outside a module in module mode (with GO111MODULE=on), its primary purpose is to build and install executables. In this configuration, there is no main module, even if only one module provides packages named on the command line. The build list (the set of module versions used in the build) is calculated from requirements in go.mod files of modules providing packages named on the command line. replace or exclude directives from all modules are ignored. Vendor directories are also ignored.

When go get is invoked inside a module, its primary purpose is to update requirements in go.mod. The -d flag is often used, which instructs go get not to build or install packages. Explicit go build or go install commands are often better for installing tools when dependency versions are specified in go.mod and no update is desired. Like other build commands, go get loads the build list from the main module's go.mod file, applying any replace or exclude directives it finds there. replace and exclude directives in other modules' go.mod files are never applied. Vendor directories in the main module and in other modules are ignored; the -mod=vendor flag is not allowed.

The motivation for the current go get behavior was to make usage in module mode similar to usage in GOPATH mode. In GOPATH mode, go get would download repositories for any missing packages into $GOPATH/src, then build and install those packages into $GOPATH/bin or $GOPATH/pkg. go get -u would update repositories to their latest versions. go get -d would download repositories without building packages. In module mode, go get works with requirements in go.mod instead of repositories in $GOPATH/src.

Rationale

Why can't `go get` clone a git repository and build from there?

In module mode, the go command typically fetches dependencies from a proxy. Modules are distributed as zip files that contain sources for specific module versions. Even when go connects directly to a repository instead of a proxy, it still generates zip files so that builds work consistently no matter how modules are fetched. Those zip files don't contain nested modules or vendor directories.

If go get cloned repositories, it would work very differently from other build commands. That causes several problems:

It adds complication (and bugs!) to the go command to support a new build mode.
It creates work for authors, who would need to ensure their programs can be built with both go get and go install.
It reduces speed and reliability for users. Modules may be available on a proxy when the original repository is unavailable. Fetching modules from a proxy is roughly 5-7x faster than cloning git repositories.

Why can't vendor directories be used?

Vendor directories are not included in module zip files. Since they're not present when a module is downloaded, there's no way to build with them.

We don't plan to include vendor directories in zip files in the future either. Changing the set of files included in module zip files would break go.sum hashes.

Why can't directory `replace` directives be used?

For example:

replace example.com/sibling => ../sibling

replace directives with a directory path on the right side can't be used because the directory must be outside the module. These directories can't be present when the module is downloaded, so there's no way to build with them.

Why can't module `replace` directives be used?

For example:

replace example.com/mod v1.0.0 => example.com/fork v1.0.1-bugfix

It is technically possible to apply these directives. If we did this, we would still want some restrictions. First, an error would be reported if more than one module provided packages named on the command line: we must be able to identify a main module. Second, an error would be reported if any directory replace directives were present: we don't want to introduce a new configuration where some replace directives are applied but others are silently ignored.

However, there are two reasons to avoid applying replace directives at all.

First, applying replace directives would create inconsistency for users inside and outside a module. When a package is built within a module with go build or go install, only replace directives from the main module are applied, not the module providing the package. When a package is built outside a module with go get, no replace directives are applied. If go install applied replace directives from the module providing the package, it would not be consistent with the current behavior of any other build command. To eliminate confusion about whether replace directives are applied, we propose that go install reports errors when encountering them.

Second, if go install applied replace directives, it would take power away from developers that depend on modules that provide tools. For example, suppose the author of a popular code generation tool gogen forks a dependency genutil to add a feature. They add a replace directive pointing to their fork of genutil while waiting for a PR to merge. A user of gogen wants to track the version they use in their go.mod file to ensure everyone on their team uses a consistent version. Unfortunately, they can no longer build gogen with go install because the replace is ignored. The author of gogen might instruct their users to build with go install, but then users can't track the dependency in their go.mod file, and they can't apply their own require and replace directives to upgrade or fix other transitive dependencies. The author of gogen could also instruct their users to copy the replace directive, but this may conflict with other require and replace directives, and it may cause similar problems for users further downstream.

Why report errors instead of ignoring `replace`?

If go install ignored replace directives, it would be consistent with the current behavior of go get when invoked outside a module. However, in #30515 and related discussions, we found that many developers are surprised by that behavior.

It seems better to be explicit that replace directives are only applied locally within a module during development and not when users build packages from outside the module. We'd like to encourage module authors to release versions of their modules that don't rely on replace directives so that users in other modules may depend on them easily.

If this behavior turns out not to be suitable (for example, authors prefer to keep replace directives in go.mod at release versions and understand that they won't affect users), then we could start ignoring replace directives in the future, matching current go get behavior.

Should `go.sum` files be checked?

Because there is no main module, go install will not use a go.sum file to authenticate any downloaded module or go.mod file. The go command will still use the checksum database (sum.golang.org) to authenticate downloads, subject to privacy settings. This is consistent with the current behavior of go get: when invoked outside a module, no go.sum file is used.

The new go install command requires that only one module may provide packages named on the command line, so it may be logical to use that module's go.sum file to verify downloads. This avoids a problem in #28802, a related proposal to verify downloads against all go.sum files in dependencies: the build can't be broken by one bad go.sum file in a dependency.

However, using the go.sum from the module named on the command line only provides a marginal security benefit: it lets us authenticate private module dependencies (those not available to the checksum database) when the module on the command line is public. If the module named on the command line is private or if the checksum database isn't used, then we can't authenticate the download of its content (including the go.sum file), and we must trust the proxy. If all dependencies are public, we can authenticate all downloads without go.sum.

Why require a version suffix when outside a module?

If no version suffix were required when go install is invoked outside a module, then the meaning of the command would depend on whether the user's working directory is inside a module. For example:

go install golang.org/x/tools/gopls

When invoked outside of a module, this command would run in GOPATH mode, unless GO111MODULE=on is set. In module mode, it would install the latest version of the executable.

When invoked inside a module, this command would use the main module's go.mod file to determine the versions of the modules needed to build the package.

We currently have a similar problem with go get. Requiring the version suffix makes the meaning of a go install command unambiguous.

Why not a `-g` flag instead of `@latest`?

To install the latest version of an executable, the two commands below would be equivalent:

go install -g golang.org/x/tools/gopls
go install golang.org/x/tools/gopls@latest

The -g flag has the advantage of being shorter for a common use case. However, it would only be useful when installing the latest version of a package, since -g would be implied by any version suffix.

The @latest suffix is clearer, and it implies that the command is time-dependent and not reproducible. We prefer it for those reasons.

Compatibility

The go install part of this proposal only applies to commands with version suffixes on each argument. go install reports an error for these, and this proposal does not recommend changing other functionality of go install, so that part of the proposal is backward compatible.

The go get part of this proposal recommends deprecating and removing functionality, so it's certainly not backward compatible. go get -d commands will continue to work without modification though, and eventually, the -d flag can be dropped.

Parts of this proposal are more strict than is technically necessary (for example, requiring one module, forbidding replace directives). We could relax these restrictions without breaking compatibility in the future if it seems expedient. It would be much harder to add restrictions later.

Implementation

An initial implementation of this feature was merged in CL 254365. Please try it out!

Future directions

The behavior with respect to replace directives was discussed extensively before this proposal was written. There are three potential behaviors:

Ignore replace directives in all modules. This would be consistent with other module-aware commands, which only apply replace directives from the main module (defined in the current directory or a parent directory). go install pkg@version ignores the current directory and any go.mod file that might be present, so there is no main module.
Ensure only one module provides packages named on the command line, and treat that module as the main module, applying its module replace directives from it. Report errors for directory replace directives. This is feasible, but it may have wider ecosystem effects; see "Why can't module replace directives be used?" above.
Ensure only one module provides packages named on the command line, and report errors for any replace directives it contains. This is the behavior currently proposed.

Most people involved in this discussion have advocated for either (1) or (2). The behavior in (3) is a compromise. If we find that the behavior in (1) is strictly better than (2) or vice versa, we can switch to that behavior from (3) without an incompatible change. Additionally, (3) eliminates ambiguity about whether replace directives are applied for users and module authors.

Note that applying directory replace directives is not considered here for the reasons in "Why can't directory replace directives be used?".

Appendix: FAQ

Why not apply `replace` directives from all modules?

In short, replace directives from different modules would conflict, and that would make dependency management harder for most users.

For example, consider a case where two dependencies replace the same module with different forks.

// in example.com/mod/a
replace example.com/mod/c => example.com/fork-a/c v1.0.0

// in example.com/mod/b
replace example.com/mod/c => example.com/fork-b/c v1.0.0

Another conflict would occur where two dependencies pin different versions of the same module.

// in example.com/mod/a
replace example.com/mod/c => example.com/mod/c v1.1.0

// in example.com/mod/b
replace example.com/mod/c => example.com/mod/c v1.2.0

To avoid the possibility of conflict, the go command ignores replace directives in modules other than the main module.

Modules are intended to scale to a large ecosystem, and in order for upgrades to be safe, fast, and predictable, some rules must be followed, like semantic versioning and import compatibility. Not relying on replace is one of these rules.

How can module authors avoid `replace`?

replace is useful in several situations for local or short-term development, for example:

Changing multiple modules concurrently.
Using a short-term fork of a dependency until a change is merged upstream.
Using an old version of a dependency because a new version is broken.
Working around migration problems, like golang.org/x/lint imported as github.com/golang/lint. Many of these problems should be fixed by lazy module loading (#36460).

replace is safe to use in a module that is not depended on by other modules. It's also safe to use in revisions that aren't depended on by other modules.

If a replace directive is just meant for temporary local development by one person, avoid checking it in. The -modfile flag may be used to build with an alternative go.mod file. See also #26640 a feature request for a go.mod.local file containing replacements and other local modifications.
If a replace directive must be checked in to fix a short-term problem, ensure at least one release or pre-release version is tagged before checking it in. Don't tag a new release version with replace checked in (pre-release versions may be okay, depending on how they're used). When the go command looks for a new version of a module (for example, when running go get with no version specified), it will prefer release versions. Tagging versions lets you continue development on the main branch without worrying about users fetching arbitrary commits.
If a replace directive must be checked in to solve a long-term problem, consider solutions that won't cause issues for dependent modules. If possible, tag versions on a release branch with replace directives removed.

When would `go install` be reproducible?

The new go install command will build an executable with the same set of module versions on every invocation if both the following conditions are true:

A specific version is requested in the command line argument, for example, go install example.com/cmd/foo@v1.0.0.
Every package needed to build the executable is provided by a module required directly or indirectly by the go.mod file of the module providing the executable. If the executable only imports standard library packages or packages from its own module, no go.mod file is necessary.

An executable may not be bit-for-bit reproducible for other reasons. Debugging information will include system paths (unless -trimpath is used). A package may import different packages on different platforms (or may not build at all). The installed Go version and the C toolchain may also affect binary reproducibility.

What happens if a module depends on a newer version of itself?

go install will report an error, as go get already does.

This sometimes happens when two modules depend on each other, and releases are not tagged on the main branch. A command like go get example.com/m@master will resolve @master to a pseudo-version lower than any release version. The go.mod file at that pseudo-version may transitively depend on a newer release version.

go get reports an error in this situation. In general, go get reports an error when command line arguments different versions of the same module, directly or indirectly. go install doesn't support this yet, but this should be one of the conditions checked when running with version suffix arguments.

Appendix: usage of replace directives

In this proposal, go install would report errors for replace directives in the module providing packages named on the command line. go get ignores these, but the behavior may still surprise module authors and users. I've tried to estimate the impact on the existing set of open source modules.

I started with a list of 359,040 main packages that Russ Cox built during an earlier study.
I excluded packages with paths that indicate they were homework, examples, tests, or experiments. 187,805 packages remained.
Of these, I took a random sample of 19,000 packages (about 10%).
These belonged to 13,874 modules. For each module, I downloaded the "latest" version go get would fetch.
I discarded repositories that were forks or couldn't be retrieved. 10,618 modules were left.
I discarded modules that didn't have a go.mod file. 4,519 were left.
Of these:
- 3982 (88%) don't use replace at all.
- 71 (2%) use directory replace only.
- 439 (9%) use module replace only.
- 27 (1%) use both.
- In the set of 439 go.mod files using module replace only, I tried to classify why replace was used. A module may have multiple replace directives and multiple classifications, so the percentages below don't add to 100%.
- 165 used replace as a soft fork, for example, to point to a bug fix PR instead of the original module.
- 242 used replace to pin a specific version of a dependency (the module path is the same on both sides).
- 77 used replace to rename a dependency that was imported with another name, for example, replacing github.com/golang/lint with the correct path, golang.org/x/lint.
- 30 used replace to rename golang.org/x repos with their github.com/golang mirrors.
- 11 used replace to bypass semantic import versioning.
- 167 used replace with k8s.io modules. Kubernetes has used replace to bypass MVS, and dependent modules have been forced to do the same.
- 111 modules contained replace directives I couldn't automatically classify. The ones I looked at seemed to mostly be forks or pins.

The modules I'm most concerned about are those that use replace as a soft fork while submitting a bug fix to an upstream module; other problems have other solutions that I don't think we need to design for here. Modules using soft fork replacements are about 4% of the the modules with go.mod files I sampled (165 / 4519). This is a small enough set that I think we should move forward with the proposal above.

jayconrod commented 4 years ago

@aarzilli

Yes. This proposal would also be that proposal.

I'm not willing to consider a change to the proxy protocol in this proposal. It's very costly.

Likewise, I wouldn't consider cloning repositories. It's wildly different than what all other commands do in module mode.

What doesn't ever have merit is "use the directives of the module providing P but error if there is a replace directive" ...

I've said this earlier:

For now, we're reporting an error so there is no ambiguity about whether replace is applied or not. Additionally, starting out with strict behavior lets us relax it later without an incompatible change.

If we find that in many cases, module authors can't feasibly remove replace directives from tagged releases, then we can apply module replacements only.

If we find that replace directives aren't usually necessary in releases but are inconvenient to remove, then we can ignore them (as other commands do).

... by virtue of the fact that a global install command should not have outcomes that depend on the directory it is run in ...

I've said this earlier:

The command proposed here does not depend on the directory it is run in.

Then why are you proposing a command with a different behavior and that the current behavior of go get should be deprecated and removed?

I explained that right below the sentence you quoted.

I don't think people add replace directives to go.mod expecting them to be ignored. I think the global install command should respect replace directives so that module authors can leave replace directives in place knowing that they will be respected.

I'm sure no one would add them if they were always ignored. But they aren't. replace directives from the main module are applied, and they're useful during development.

replace directives outside the main module are ignored because they'd inevitably lead to conflicts. If the module example.com/Main requires example.com/A and example.com/B, and those modules both require and replace example.com/C with different versions, the author of example.com/Main would not be able to build anything.

I know the current behavior isn't always what people expect from replace, but I think that's largely because we haven't done an adequate job documenting it. We're making progress on that though.

I'm the author of a tool, I send a PR to one of my dependencies, unfortunately the author has taken a long vacation to Mars and the PR will be merged 6 months from now. In the meantime I fork the dependency and add a temporary replace directive. People try to install my tool using go install but it doesn't work, because replace directives aren't respected. How am I not a module user, and how isn't this taking power away from me and giving it to the Mars vacationer?

I'm interested in the specifics of situations like this. Mars is hyperbole of course, but why can't the PR be merged? Is the module permanently abandoned, in which case it's better to do a hard fork? Is the PR a bug fix or feature request? Could you work around the bug without replace in release versions until the bug is fixed? Does the author disagree with the PR? Is there a licensing reason you're unable to submit a PR?

Based on the modules I looked at when building the proposal's appendix, replace is not usually strictly necessary, or it wouldn't be necessary in the future when other proposals like #36460 land. But I think we need more data here to make a decision in either direction.

How would respecting replace directives break it?

This is explained in "Why can't module replace directives be used?".

It's because it causes inconsistencies and likely incompatibilities for modules that depend on modules providing tools. Those modules rely on replace directives in other modules being ignored.

Suppose I have a module example.com/Main that uses a static analyzer in the module example.com/analyze. I add a requirement to my go.mod, and in a build script, I have a command like:

go run example.com/analyze/cmd/analyzer -- ./...

Suppose I found a bug in a package that analyzer depends on in golang.org/x/tools, and I submitted a fix. While I'm waiting for that to be merged, I've added the line below to my go.mod:

replace golang.org/x/tools => github.com/jayconrod/tools v1.0.0-bugfix

Separately, the author of analyzer submitted a PR adding a new feature to golang.org/x/tools, so they have this replace directive in their go.mod.

replace golang.org/x/tools => github.com/analyzer/tools v1.1.0-feature

The package example.com/analyze/cmd/analyzer doesn't build without that feature, which means I can't use my go run script. My build is broken.

When I complain, the author of analyzer says, "Just use go install example.com/cmd/analyzer@latest" (i.e., this proposal except replace directives are applied). But then I have no way to build analyzer with my bug fix. Effectively, this proposal (with replace directives) took power from me, a module user, and gave it to the module author.

jayconrod commented 4 years ago

@aarzilli

What doesn't ever have merit ...

... I'm not sure what other commands you think are relevant.

Just want to add, I think your tone in this discussion (and these two phrases in particular) has been unnecessarily argumentative, verging on disrepectful. Please keep the code of conduct in mind.

jayconrod commented 4 years ago

@thepudds

I've added a "Future directions" section that recaps the discussion here. Please let me know if I've missed anything.

gopherbot commented 4 years ago

Change https://golang.org/cl/258297 mentions this issue: cmd/go: error if -modfile used with 'go install pkg@version'

rsc commented 4 years ago

To summarize the discussion a bit, many questions have been asked that are now answered in the "Appendix: FAQ" in the top comment above.

The main point of contention remaining seems to be whether or not to apply replace directives. It can seem attractive to do so at first, but the result is various inconsistencies/paradoxes with go command execution and ecosystem fragmentation. My understanding is that this proposal has been discussed at the monthly tools meetings as well and that the general consensus there is that we should move ahead with replace disallowed (rejected).

Note that disallowing replace means that we can introduce semantics for replace at some point in the future. Making a decision now locks us in to something that will probably not be right. @bcmills also has some cleanup of replace planned for Go 1.17, so it would be good not to preempt that. Rejecting replace for now is the conservative behavior.

Other than replace, does anyone have any objections to accepting this proposal?

mtibben commented 4 years ago

165 used replace as a soft fork, for example, to point to a bug fix PR instead of the original module.

242 used replace to pin a specific version of a dependency (the module path is the same on both sides).

77 used replace to rename a dependency that was imported with another name, for example, replacing github.com/golang/lint with the correct path, golang.org/x/lint.

30 used replace to rename golang.org/x repos with their github.com/golang mirrors.

11 used replace to bypass semantic import versioning.

167 used replace with k8s.io modules. Kubernetes has used replace to bypass MVS, and dependent modules have been forced to do the same.

111 modules contained replace directives I couldn't automatically classify. The ones I looked at seemed to mostly be forks or pins.

The modules I'm most concerned about are those that use replace as a soft fork ... other problems have other solutions that I don't think we need to design for here

I'd love to see an expansion on this - what is the recommended course of action for each of these scenarios if it's not using replace. And if this is not the place to talk about that, where might this discussion take place?

aarzilli commented 4 years ago

@rsc

The main point of contention remaining seems to be whether or not to apply replace directives. It can seem attractive to do so at first, but the result is various inconsistencies/paradoxes with go command execution and ecosystem fragmentation.

I think that the most important inconsistency is the one that would exist between running go build within the module that provides the package being installed and go install P. Since running build/test/etc within the context of the module that's being developed is the way programs are developed normally this would effectively introduce a different build mode (where replace directives are errors).

rsc commented 4 years ago

I think that the most important inconsistency is the one that would exist between running go build within the module that provides the package being installed and go install P. Since running build/test/etc within the context of the module that's being developed is the way programs are developed normally this would effectively introduce a different build mode (where replace directives are errors).

The different build mode effectively already exists. Today if you run "go build P" or "go install P" from any module other than P itself, the replaces are silently ignored, meaning that you get different results from the build inside the module. That different build is not what's expected for this one-shot install command; rejecting the replacements instead of silently ignoring them helps make that clear.

It seems to me that the answer is to remove any local development-only replaces before tagging a release version. The removal doesn't even have to happen on the main branch: do the remove in detached HEAD mode & tag.

mvdan commented 4 years ago

I think the disconnect here is that some people are arguing that using replace directives in releases is sometimes a necessity, but in my opinion that debate needs to happen in another thread.

aarzilli commented 4 years ago

@rsc

The different build mode effectively already exists. Today if you run "go build P" or "go install P" from any module other than P itself, the replaces are silently ignored, meaning that you get different results from the build inside the module.

I undestand but I don't think this is a useful or expected behavior and expanding it to the global install command would make a rare edge case into a common occourence. Why is it the case that go build P silently ignores replace directives when it's run outside of any module but in module mode?

It seems to me that the answer is to remove any local development-only replaces before tagging a release version

But what if I want my release to use replace directives?

@mvdan

I think the disconnect here is that some people are arguing that using replace directives in releases is sometimes a necessity, but in my opinion that debate needs to happen in another thread.

Absolutely agree on the first part but I don't think it this discussion can be divorced from this proposal since it amounts to a soft deprecation of replace directives.

bcmills commented 4 years ago

Why is it the case that go build P silently ignores replace directives when it's run outside of any module but in module mode?

That is not the case. As of Go 1.15, go build of a package from a module fails explicitly when run outside of any module but in module mode.

$ go version
go version go1.15.2 linux/amd64

$ export GO111MODULE=on

$ go env GOMOD
/dev/null

$ go build golang.org/x/tools/cmd/bundle
cannot find module providing package golang.org/x/tools/cmd/bundle: working directory is not part of a module

bcmills commented 4 years ago

@mtibben, the golang-nuts mailing list or the #modules channel on the Gophers Slack would be good starting points for that discussion. (At some point we should probably also write a blog post on fixing module dependencies, although the general formula in nearly all cases is “fix upstream, then upgrade to incorporate the fix”.)

aarzilli commented 4 years ago

That is not the case. As of Go 1.15, go build of a package from a module fails explicitly when run outside of any module but in module mode.

Interesting. If that's the case a global install command that respects replace directives wouldn't be any more inconsistent with the behavior of other commands than one that respects them but always behaves as if there was no current module. What am I missing?

jayconrod commented 4 years ago

Interesting. If that's the case a global install command that respects replace directives wouldn't be any more inconsistent with the behavior of other commands than one that respects them but always behaves as if there was no current module. What am I missing?

The global install command that's currently available outside a module is go get pkg@version. go install pkg@version is meant to be similar to that. If the module named on the command line satisfies the constraints listed in the proposal, then go install pkg@version will build the same thing as go get pkg@version. The only difference is that go install pkg@version could be run from anywhere, while go get pkg@version must be run outside a module or it will use and update the go.mod in the module's directory.

So we're not introducing a new build mode. We're making an existing build mode accessible through another command, with some constraints to avoid confusion.

rsc commented 4 years ago

Based on the discussion above and the responses to my question last week, this seems like a likely accept.

clausecker commented 4 years ago

It would be kinda cool to have similar behaviour for go test. I some times want to run benchmarks on different versions of a module to check for performance or other regressions and being able to do so without having to manually check out the repository would be great. Similarly, I believe there is a value in being able to run tests and benchmarks for dependencies (to make sure they pass) without having to check out the sources explicitly.

rsc commented 4 years ago

No change in consensus, so accepted.

rsc commented 4 years ago

(And already implemented.)

gopherbot commented 4 years ago

Change https://golang.org/cl/266360 mentions this issue: cmd/go: print deprecation messages for 'go get' installing executables

gopherbot commented 3 years ago

Change https://golang.org/cl/270980 mentions this issue: cmd/go/internal/modload: remove SetBuildList

gopherbot commented 3 years ago

Change https://golang.org/cl/285452 mentions this issue: content/static/doc: document 'go install pkg@version'

gopherbot commented 3 years ago

Change https://golang.org/cl/332569 mentions this issue: cmd/go/internal/modload: fix an apparent typo in the AutoRoot comment

Barmem commented 2 years ago

How would i update installed executables?

philipwhiuk commented 2 years ago

Why wasn't the removal of go get a major version change. What's the point of 1. if Go maintainers are happy to make breaking changes in a point release. It's fundamentally broken builds that don't use modules.

mvdan commented 2 years ago

@philipwhiuk please read https://go.dev/doc/go1compat:

Finally, the Go toolchain (compilers, linkers, build tools, and so on) is under active development and may change behavior. This means, for instance, that scripts that depend on the location and properties of the tools may be broken by a point release.

Of course they are still careful when breaking the tooling, but it's allowed by the compatibility guarantee.

philipwhiuk commented 2 years ago

That "compatibility guarantee" is a big "yeah we don't care about semver for tools, good luck" isn't it.

clausecker commented 2 years ago

@philipwhiuk Again, the compatibility promise is about the language and libraries, not the tooling. It's more about being able to write code that will compile in the future. The toolchain is disposable and can change, the language is stable.

Xe commented 2 years ago

I know this is probably the wrong place to put this and that this feedback is likely going to be ignored, but this whole "the tooling is exempt from the compatibility promise" thing kind of feels like a cop-out. I understand the intent of this, as you use Go in production places then you will learn more things and realize what mistakes were made; however the community starts to standardize and expect the behaviour of commands to stay consistent unless there is a major version change. This violation of expectations is why you see people make more heated or angry comments. It feels like the tool changed out from under them and now they have to re-learn the tool.

I'd be willing to argue that the semantics of how commands like go get work is probably more important than the standard library being stable. Something like go fix can correct breaking changes in the standard library, but there is no go fix to correct the changes in how people install commands or the documentation people have written about how to install various tools written in Go.

Again, I get the point of why the tooling isn't behind the compat promise. It allows you to make the tooling better and fix behaviours that were outright bugs, but overall I'm really not sure if continuing to keep Go at 1.x post-modules was really a good idea. I know that politically making a "Go 2" is a difficult thing, but the kinds of changes that modules have given developers at nearly every level of the stack really seems like it should have been a candidate for being released as "Go 2". It's sad that this is politically difficult though, because "Go 2" has been lauded as a magical fairy unicorn thing where the standard library is going to be fixed forever or whatever. If anything what we have now may as well be "Go 2" because the Go I started learning near a decade ago is almost a completely different language than the Go I use today.

philipwhiuk commented 2 years ago

@philipwhiuk Again, the compatibility promise is about the language and libraries, not the tooling. It's more about being able to write code that will compile in the future. The toolchain is disposable and can change, the language is stable.

It barely matters if the code still "compiles" if I can't actually get it to compile by running the same commands. Code that works on 1.13.4 will have to be rewritten to use modules most likely because there's no explanation of how to do what I was doing with go get in modern Golang. The compile promise is worthless on it's own.

mvdan commented 2 years ago

I think we're getting off topic for this thread, especially as it's been closed for a while. I would suggest to give https://github.com/golang/go/issues/37755 a read, which was a proposal to actually keep the pre-modules workflow working forever. For any other meta discussion about the backwards compatibility guarantee, I would personally suggest https://groups.google.com/g/golang-dev or perhaps a new issue if you have a specific proposal in mind.

mihaiav commented 2 years ago

After I've "upgraded" to go 1.18 another go command (this time go install) beame totally useless and my all my builds fail. All the related issues seem to be closed. Better remove all the go commands so that can go back to good old make files. At least we know what we are dealing with.

briantopping commented 1 year ago

I was led here via https://golang.org/doc/go-get-install-deprecation.

Didn't realize this was going on at all. Whether it was sloppy/lazy or whatnot, I would typically check out sources on projects I wanted in GOPATH via GOMODULE111=off go get XXX. A one-liner that checked out the sources. Reading around today, I see that I should have been using go get -d. Okay.

Created a new machine today and blindly installed the latest. My muscle memory doesn't work. Wasted a couple hours to realize it wasn't because some magic was lost in my GOPATH, the feature was removed.

Is there a new way to populate $GOPATH/src without creating directories, changing to them and manual git clone?

I guess I always considered that a feature and not an edge case?

mvdan commented 1 year ago

@briantopping https://github.com/golang/go/issues/31529 tracks what would be the replacement for GO111MODULE=off go get in terms of VCS cloning into a useful directory.

fubss commented 1 year ago

For those who like me:

need to upload the source code of the project to the ~/go/src directory;
are not comfortable with git clone; 1) install an additional version of go where else one could use go get (I installed 1.18.10): 1.1) go install golang.org/dl/go1.18.10@latest 1.2) ~/go/bin/go1.18.10 download 1.3) ~/go/bin/go1.18.10 version 2) export GO111MODULE=off 3) ~/go/bin/go1.18.10 get -d [desired_project_repo_here]

golang / go