cmd/go: modify the Go toolchain to work without GOPATH

[rasky]

Over the years, when helping people approaching existing Go projects (colleagues, friends, etc.) the number one problem is of course that they don't know about GOPATH, they don't have one configured, and they expect to be able to clone an existing project wherever they want on the disk, and be able to build it. I think the problem statement is clear and the problem is well known.

With the introduction and adoption of the vendoring folder, most Go applications do not even require go get to run, after the initial clone: all the source code required to compile them is already available in the source tree; it's just that the Go tool doesn't know where to look if GOPATH is not defined.

Instead of suggesting a default GOPATH (which does not solve the fact that people will need to find out about it and use it somehow), I suggest that we change the Go toolchain in multiple steps (each one can be independently released as an experiment):

• STEP 1: If GOPATH is not set, the Go tool will use any existing ./vendor directory as it was part of the GOPATH. (I think of this as defaulting GOPATH/src to ./vendor). At least go build and go run are guaranteed to work. Anything else, like go install and go get, would fail with an error pointing to the GOPATH documentation.
• STEP 2: if GOPATH is not set, the Go toolchain would be updated to use $HOME/.gopkg (or similar) as pkg directory. This would allow other commands to work (e.g.: go build -i or go install).
• STEP 3: if GOPATH is not set, go get (or the new package manager, if/when it gets released) would be modified to download packages directly into ./vendor

As shown in Step 2, I think this proposal would also interact well with the existing proposal/discussion of dropping pkg in favour of a hidden cache directory, as that would move things further into the direction of not needing GOPATH anymore for a whole class of development; in fact, this functionality could be prototyped in this specific scenario of "GOPATH not defined", where it would make sense to do it without facing the bigger discussion of deprecating pkg for a normal GOPATH scenario.

As shown in Step 3, I also think that this might interact well with the new package manager. I saw many people requested that the new package manager ought to be able to download dependencies directly into the vendor folder (like govendor and glide can do); if this functionality lands in the new package manager, it would be a perfect default for the "GOPATH not defined" scenario described here. Alternatively, it might make sense to discuss modifying go get to have this behaviour.

I think this proposal is incremental, backward-compatible, doesn't affect existing users that successfully use GOPATH, and allow beginners to approach existing Go codebases without stumbling too soon into GOPATH.

[lloeki]

"I have a project in a random place, all deps are in vendor/, what is GOPATH, I want to build this" while not breaking go get for that project. It also has the nice side effect you mentioned of ignoring the rest of the GOPATH.

There are two key points, really, here:

• "Ignoring the rest of the GOPATH": this is not just a nice side effect, as defaulting GOPATH to ~/go plus vendoring makes many things in the above sentence work already, save from the next point, which is the only thing that forces the user to get what GOPATH is
• getting the full import path for the project, which is obtained from:
  • the dir path from within GOPATH
  • metadata that contains the name (IMPORT_PATH in Makefile, arguably symlink in my case)
  • possibly git remote get-url origin and reverse the transformation that go get does already to find the repo path from the import path

The user suffers from not reading the manual, which can be fixed by having the user reading the manual

Then the user could have been reading about GOPATH ;)

this one can traverse the directory searching for the metadata files.

Which metadata files? Go currently has none. Third party Go tools may have (such as a depfile/lockfile) but I don't think it'd be sane to rely on that, and @FiloSottile suggested something like an .IMPORT_PATH file extracted from the Makefile. Then again, a repo could have multiple projects each with its own vendor, while its .go_import_path or something would hint Go at the "import name" of the repo root in order to map "filesystem space" to "import space" and back (I like this).

Again, I'm not trying to solve all the problems from the release, just enough of them to have a decent (not even good) solution to build on top of without too much disruption of the current workflows.

Obviously I'm playing devil's advocate here :) but there were some points I found unsatisfying and sufficiently disrupting so I wanted to highlight them.

I'm specifically talking about the real world where projects are separated. It doesn't have to solve all the problems, it just has to solve enough of them. This also can be addressed later on.

What I'm suggesting though is that both behaviors can be had, the no-GOPATH one we're discussing and the GOPATH one as it works currently can coexist because they're both useful in different real-world contexts. The no-GOPATH one "merely" reverts the default GOPATH being ~/go into something arguably more useful, thanks to the introduction of vendor and dep, and the value of the GOPATH env var toggles between both. I don't see why there would be a need to sacrifice the current GOPATH way for no-GOPATH to work, although the former may benefit from some enhancements coming from the latter. Obviously shielding such a change of behavior change behind a NOGOPATHEXPERIMENT toggle is indicated to iterate as @dlsniper suggests.

[rivy]

What happens if they are in a subdirectory and run source ../../.env ?

Not to derail a productive discussion about removing the need for $GOPATH (👍)... but here is a version of .env that can be used / source'd portably within a project.

#!/bin/sh
# setup GO environment for project
eval $(
  dir=$(dirname -- "$0")
  dir_abs=$(unset CDPATH; cd -- "${dir}" && pwd -L)
  echo "export GOPATH=\"${dir_abs}:${dir_abs}/vendor\";"
  echo "export PATH=\"${dir_abs}/bin:${PATH}\";"
)

It works for the standard *nix shells and has no dependency on the current working directory.

The eval/echo wrap is used to avoid unintended clobbering of parent environment variables.

cc: @DanielHeath , @davecheney

[sbinet]

@rivy I myself am fond of direnv for these sort of things. (bonus point: it's written in Go itself)

[ascotan]

if demolib has any other dependencies they should also be downloaded into $PWD/vendor

That's the sticky widget isn't it?
Transitive dependency management is the bane of many systems that have come before (maven, npm) In theory A can use B and C (release 1.2), and B can also use C but a just different version of it (HEAD). Oops.

I guess the question is - can/should a Go application be allowed to use multiple versions of a library? It seems like the answer should be no, but then how do you solve that? This seems highly related to the behavior of the /vendor folder.

Poking around with gvt, it looks like they simply ignore any subvendor folders. The problem with this is that you might get unexpected behavior if a dependency is dependant on a similar but slightly different version that has the same method signatures.

crash() bool { // version 1.1
  return true
}

crash() bool { // HEAD
  return false
}

[dlsniper]

That's the sticky widget isn't it? Transitive dependency management is the bane of many systems that have come before (maven, npm) In theory A can use B and C (release 1.2), and B can also use C but a just different version of it (HEAD). Oops.

I'm fully aware of the diamond problem, thanks for the oops right there :)

Yes, but if you also read what I've wrote, it shouldn't be up to go get to solve that. The initial version can do whatever the current go get does so the problem is solved, the behavior is the same.

Should we also discuss about the complexity of resolving all dependencies in automatic way? Let's do that in the next section.

I guess the question is - can/should a Go application be allowed to use multiple versions of a library? It seems like the answer should be no, but then how do you solve that?

In my opinion, no. That's a can of "goodies" that it's best left unhandled by automated tools in a "smart" way.

Again however, this shouldn't be a problem that go get shouldn't solve, at least not in an initial release. This should be solved via the metadata file(s) present in the repository which will tell the tool go get tool to fetch the exact versions that you need. Will this tool be the same as go get? Maybe, I don't know.

We also have two problems here:

• people version their repositories and follow semver religiously
• people version their repositories and don't follow semver even remotely religiously

If the first thing happens, then you are lucky. But it's highly likely that a similar situation as the one you are describing, where a small change that technically doesn't effect the API but the behavior of the package.

For this and the latter case where a user doesn't follow semver practices you really have a single solution: ask the user to solve the conflict issue. It's really the only way to guarantee to the user that the package that is in a version conflict state can be used. Sure someone might say: but we can write tools for that. I agree, at some point we might have better tools that detect these things, but are you willing to wait until they get created and tested and proved to never be wrong?

We could argue for the third case where people don't version their repos but then we can also say: just fetch the HEAD and let the users complain about the missing versioning. This will create enough pressure on the maintainers to start being better at maintaining their packages, which is totally ok.

This seems highly related to the behavior of the /vendor folder.

It's a problem today for get as well, with the GOPATH only, not just the vendor folder.

I'm sure there's someone that will soon point out even more issues, hypothetical or real, but again, I urge you, please take a couple of steps back and ask yourself:

• What's important? Having a way to work with an application focused workflow? Or keeping compatibility with GOPATH?
• (or in a rewording of the above): - What's important? Having a way to get the dependencies for your application in the vendor folder? Or having a way to have go get compatibility for applications?
• What's important? Having a way to cover 80%-90% of the cases out of the box and provide the users a way to reliably say go get ./... in a project root directory and fetch the dependencies at their expected, predefined versions? Or solving a lot of the corner cases that appear along the complex subject that versioning is, in an automatic way?

I'm not saying effort shouldn't be put into trying to solve as many issues as possible and make the life of the users as good as possible. But please remember that Go itself doesn't have this approach to solve problems and we can approach this in an incremental manner while still offering something stable and usable to the users at all times.

[lloeki]

Coming from the ruby world and having dabbled with node, my personal practical experience as well as theoretical belief totally concurs that using a single version of a lib is the way to go. Go has the added benefit that an incompatible version has a greater chance of being detected statically at compile time compared to dynamic languages which have to rely on tests. If something arises from detecting an incompatible version whether through semver, static analysis, or tests, then it means some mess needs to be sorted out, and the node way is just trying to sweep it under the rug, only adding complexity to the table when (when!) you have to untangle object lifecycle through multiple versions of the same lib that may have subtly different expectations of data structures as well as behaviors.

[dlsniper]

@rsc this is the issue we've talked about at GopherCon.

I believe that the proposal to have a "virtual GOPATH" created based on the vendor/ directory presence (and probably the golang/dep manifest/lock files).

In my simplistic view, this could be done for Go 1.10 but I defer to your judgement if this could be approached in the proposed format, or otherwise, by then.

Thank you.

[rasky]

I think that it would be a good time to revisit this proposal that was put on hold 15 months ago (/cc @bradfitz @rsc).

My experience hasn't changed much since I opened this proposal, and the default GOPATH hasn't helped a bit, as I was describing in this comment. Given also the new cache (that was created outside GOPATH, and basically cancels the needs that were described in STEP2 in the proposal), we're really close to being able to let users play with a Go project (with vendored dependencies) without having a GOPATH set, that is STEP1 in my original proposal.

[bradfitz]

Unholding.

[target-san]

What I'd suggest may have been already proposed. Anyway

1. Make go tools respect and use per-user directory, like $HOME/.gocache by default
2. All packages pulled via go get land automatically into $GOCACHE/src/<packagepath>. Some versioning prefix or suffix may be added to package's path - disputable.
3. To build project, user needs to pull its sources and then go build, which will involve:
   1. go dep ensure automatically and place all deps into source cache, as defined above
   2. Actual go build which will look for dependencies in cache folder, not in $GOPATH/src or ./vendor
4. Make $GOPATH deprecated
5. go install may actually deploy binaries to $GOCACHE/bin for convenience
6. ...

This way, each project is self-contained and can be built right from sources, without managing dependencies and their locations manually

[kardianos]

This issue will be resolved should vgo or similar solution be adopted. The need for a GOPATH env var goes away when adding a go.mod file with the module root path.

[stevenroose]

@target-san

3. go dep ensure automatically and place all deps into source cache, as defined above
4. Actual go build which will look for dependencies in cache folder, not in $GOPATH/src or ./vendor

Different projects can have different versions for packages. I would strongly suggest to have dep ensure still use a project-specific cache: either ./vendor/ or something like $GOCACHE/build/$PROJECT/. This way, project-specific package sources go first, and then the regular Go cache (current Go path) is used.

[mathstuf]

Different projects can have different versions for packages.

Could the cache be keyed on the name and version of the package rather than just the name?

[DanielHeath]

Given that they're just files (and therefore could have been modified), I'd suggest keying on the hash of the package source instead of any surrogate key.

[devnev]

Hey all,

My experience covers work both inside a mono-repo and with individual libraries, with either fully vendored dependencies or with all the dependencies already inside the repository. The extra GOPATH structure has added little value while making it annoying to set up and manage clones of the repository under development.

I’ve been experimenting on implementing some of the changes necessary to work without a GOPATH (in a repository with fully vendored dependencies) to see what it might entail. As far as I can tell, this change is comparably small and localised, although I didn’t implement a complete solution.

A possible way to allow users to use the build tools without a GOPATH might be to walk up the filesystem from the starting directory until a source file with a specific directive (e.g. //go:prefix github.com/foo/bar) is found. If this is the case, the directory containing that source would be the root of a GOPATH-less source tree. This tree is placed in the import namespace at a prefix specified as part of the directive.

I think this allows new GOPATH-less-capable repositories to be fully compatible with older toolchain versions (the prefix directive can be ignored if it is located in a directory below a GOPATH root). Also, because the prefix scan occurs just once and picks the first prefix it finds, ambiguity from having multiple such prefixes is avoided and minimal build overhead is added. Because the source tree behaves as if placed inside a virtual GOPATH root, most of the build behaviour continues to work unchanged, and import statements do not need any adjustment.

Disregarding the particular details though, it would be great to see some movement on this issue.

[kardianos]

@devenv and any others: @rsc is currently working on a proposal that will not only take care of this issue (completly remove the need for GOPATH), but also take care of making imports version aware.

His papers on the subject: https://research.swtch.com/vgo His placeholder proposal: https://golang.org/issue/24301

Right now his proposal issue is empty and used as a placeholder for drafting the proposal text CL.

[rsc]

We're certainly moving toward "modify the Go toolchain to work without GOPATH", so accepting that idea, although not the details in the top post above. It will take a few releases to get there. The steps are

1. Add modules.
2. Stop writing anything to GOPATH/pkg (aka #4719; implies disallowing GOCACHE=off or at least making that very slow).

We're not ready to do 2 yet.

[rsc]

This is really just blocked on #24301 and #4719. After they're done, we get this for free. Closing as a duplicate of #4719.

[rsc]

Duplicate of #4719.