cmd/go: modify the Go toolchain to work without GOPATH

rasky commented 8 years ago

Over the years, when helping people approaching existing Go projects (colleagues, friends, etc.) the number one problem is of course that they don't know about GOPATH, they don't have one configured, and they expect to be able to clone an existing project wherever they want on the disk, and be able to build it. I think the problem statement is clear and the problem is well known.

With the introduction and adoption of the vendoring folder, most Go applications do not even require go get to run, after the initial clone: all the source code required to compile them is already available in the source tree; it's just that the Go tool doesn't know where to look if GOPATH is not defined.

Instead of suggesting a default GOPATH (which does not solve the fact that people will need to find out about it and use it somehow), I suggest that we change the Go toolchain in multiple steps (each one can be independently released as an experiment):

STEP 1: If GOPATH is not set, the Go tool will use any existing ./vendor directory as it was part of the GOPATH. (I think of this as defaulting GOPATH/src to ./vendor). At least go build and go run are guaranteed to work. Anything else, like go install and go get, would fail with an error pointing to the GOPATH documentation.
STEP 2: if GOPATH is not set, the Go toolchain would be updated to use $HOME/.gopkg (or similar) as pkg directory. This would allow other commands to work (e.g.: go build -i or go install).
STEP 3: if GOPATH is not set, go get (or the new package manager, if/when it gets released) would be modified to download packages directly into ./vendor

As shown in Step 2, I think this proposal would also interact well with the existing proposal/discussion of dropping pkg in favour of a hidden cache directory, as that would move things further into the direction of not needing GOPATH anymore for a whole class of development; in fact, this functionality could be prototyped in this specific scenario of "GOPATH not defined", where it would make sense to do it without facing the bigger discussion of deprecating pkg for a normal GOPATH scenario.

As shown in Step 3, I also think that this might interact well with the new package manager. I saw many people requested that the new package manager ought to be able to download dependencies directly into the vendor folder (like govendor and glide can do); if this functionality lands in the new package manager, it would be a perfect default for the "GOPATH not defined" scenario described here. Alternatively, it might make sense to discuss modifying go get to have this behaviour.

I think this proposal is incremental, backward-compatible, doesn't affect existing users that successfully use GOPATH, and allow beginners to approach existing Go codebases without stumbling too soon into GOPATH.

davecheney commented 8 years ago

If you don't want to use $GOPATH, there's always gb :)

But seriously, in my experience the problem newcomers have with $GOPATH is not its prescribed namespace layout, but the fact that they have to choose a location.

Using the go tool does have a learning curve, that is undeniable, but I believe the go tool, and the conventions of laying out source in the $GOPATH form aren't hard to learn, and once people understand the rationale behind the $GOPATH layout there is general approval.

My proposal in #17262 seeks to defer the additional cognitive load of choosing a $GOPATH value by choosing a reasonable one for the user. Users still need to understand how $GOPATH works, because that is also how import statements work in Go code because the compiler interprets an import statement as a path rooted at some location, that being a segment of the users' supplied $GOPATH. But that additional learning can take place after they've been able to write small Go programs and go get code from others.

rasky commented 8 years ago

But seriously, in my experience the problem newcomers have with $GOPATH is not its prescribed namespace layout, but the fact that they have to choose a location.

I respectfully disagree. I never had somebody complain to me that choosing GOPATH was difficult per-se (as in, how to call the directory). Their main objection is that there's no good GOPATH location because they usually have projects layed out as ~/Sources/prjname, and there is no good way of matching this layout with that imposed by GOPATH. Choosing a default GOPATH would swap one confusing error message with another. If you go with defaulting GOPATH with something, they would still have to realise that:

There is a thing called GOPATH
That thing has a default (that might point to a directory they don't even have)
Checking out their project of choice in $GOPATH/projectname will still not work
Checking out their project of choice in $GOPATH/src/projectname might still not work (in case the project refers to one of its subpackage as github.com/whatever/projectname/foo).

My proposal removes all four hurdles above; it erases the whole cognitive load associated with GOPATH for the purpose of a clone+modify+push+pull-request cycle, and thus lowers the barriers of entry much more.

Also, if we look at a modern Go application, with all its dependencies correctly vendored, I really can't see what GOPATH is buying to a user that's just trying to edit and recompile such application. Every source file that must be built is already there; it looks overly aggressive to ask them to shuffle their directories around and set environment variables just to be able for the Go tool to find a library that is stored in the official vendor subfolder. It did make more sense when they had to go get to fetch the dependencies, but that step is not so common anymore.

dmage commented 8 years ago

But which import path supposed to use for the project's packages if there is no GOPATH?

sdwarwick commented 8 years ago

The GOPATH issue has been a real stumbling block. We have multiple projects for multiple clients, each project uses a multitude of different languages and frameworks. They are all in different directories, different volumes etc.. We want all code associated with a specific project to be separately managed. It has been very difficult to see how the GOPATH architecture supports this. It is certainly not documented in the standard "getting started" information.

rasky commented 8 years ago

@dmage I think it should just be "." or whatever it is called the top-level path in the import hierarchy. This would mean that, to work without GOPATH, projects including sub-packages will have to refer to them with a relative path like "./foo". I think it's a reasonable compromise for projects that wish to build without GOPATH.

(a possibly crazy idea is to also use canonical import path for this, that is a way of teaching the Go toolchain the path of the current project being built in a no-GOPATH scenario)

ngrilly commented 8 years ago

Their main objection is that there's no good GOPATH location because they usually have projects layed out as ~/Sources/prjname, and there is no good way of matching this layout with that imposed by GOPATH.

As I work on several projects using Go alongside otherside programming languages, this is exactly my experience.

kokes commented 8 years ago

Since #17262 is going forward, the "if GOPATH isn't set, use '.'" can no longer apply. But a slight change to "if outside the GOPATH, use '.'" would do the trick - mimicking gb's behaviour and allowing people to just download/copy a piece of code and compile it then and there (possibly with a friendly warning message, that the user is outside the GOPATH).

Then again, this sounds an awful like #12488, which was declined.

adg commented 8 years ago

Regarding the resolution of #17262: while this proposal takes advantage of an unset GOPATH, I don't see that as a key element. It is still possible to take advantage of a ./vendor outside an explicit or implicit GOPATH.

rasky commented 8 years ago

Well, yes, I guess it's possible to adapt a part of this proposal with a default GOPATH (though STEP3 probably will be impossible to implement, for instance), but I think this proposal (e.g. in this comment: https://github.com/golang/go/issues/17271#issuecomment-250327512) makes a point that a default GOPATH isn't really helping, and instead defines a different, more comprehensive workflow to help beginners work with Go without having to learn about GOPATH from the get go. If the decision on #17262 is final (which I consider very unfortunate, as I think this proposal is a superior and more comprehensive solution to the problem statement), I'll have to revisit this proposal.

justinfx commented 8 years ago

I have had to deal with the problem that this proposal offers to solve. That being, making it easy for the non-go developers at my company to be able to check out a Go project to a random location and build it.

Because I can't expect every developer to have a GOPATH set up, or to clone to a specific location, I have needed to design Go logic into our waf-based build system. This has to try and do extra smarts such as forming a GOPATH on the fly if it can. Telling them to use gb is not an option for multiple reasons. They would need a special tool as a non go developer, and gb takes its own approach to the vendor directory (last I checked) in having vs not having a src subdirectory.

It would be fantastic if the standard go tool could offer sane default behavior for trying to build a project that is outside gopath, before failing. If it at least tried to assume the project is self contained with a vendor directory, it would have a chance to build.

Isn't this following the same goal as #17262 in having default behavior that makes it easier for beginners?

davecheney commented 8 years ago

The go tool requires GOPATH, it's built into its design and backed by its interpretation of the import statements. They are inseperable.

What is it specifically that you don't like about gb? It seems like its design of a self contained project structure is what you need. How can I help make gb a better fit for your requirements?

On Wed, 26 Oct 2016, 05:42 Justin Israel notifications@github.com wrote:

I have had to deal with the problem that this proposal offers to solve. That being, making it easy for the non-go developers at my company to be able to check out a Go project to a random location and build it.

Because I can't expect every developer to have a GOPATH set up, or to clone to a specific location, I have needed to design Go logic into our waf-based build system. This has to try and do extra smarts such as forming a GOPATH on the fly if it can. Telling them to use gb is not an option for multiple reasons. They would need a special tool as a non go developer, and gb takes its own approach to the vendor directory (last I checked) in having vs not having a src subdirectory.

It would be fantastic if the standard go tool could offer sane default behavior for trying to build a project that is outside gopath, before failing. If it at least tried to assume the project is self contained with a vendor directory, it would have a chance to build.

Isn't this following the same goal as #17262 https://github.com/golang/go/issues/17262 in having default behavior that makes it easier for beginners?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17271#issuecomment-256134306, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAcAwmMhO6OiiB5eMHxl36Y-d_we155ks5q3k2ogaJpZM4KJZPO .

justinfx commented 8 years ago

What is it specifically that you don't like about gb? It seems like its design of a self contained project structure is what you need. How can I help make gb a better fit for your requirements?

It is the very fact that gb diverges from the official standard, that being the go tool. Problem is that I cannot force a developer to build their Go projects with gb, so that I can then assume I can use gb to build it later, in our build system layer. The most likely path is that a developer will use the official go tool and structure their project (including the vendor directory layout) to match. Then I have to make sure any other developer can clone that project and build it, regardless of where they clone it. To use gb, I would have to make it a standard that everyone follows.

davecheney commented 8 years ago

On Wed, 26 Oct 2016, 06:26 Justin Israel notifications@github.com wrote:

What is it specifically that you don't like about gb? It seems like its design of a self contained project structure is what you need. How can I help make gb a better fit for your requirements?

It is the very fact that gb diverges from the official standard, that being the go tool.

That's fair, but I would suggest that the chances of the Go tool making GOPATH optional are about the same likely hood as gb acquiring some official status.

The important thing for me when I built gb was the realisation that the go tool isn't the lanaguge or the compiler, it's just a driver that feeds files to the compiler in the correct order, so in that respect gb is a peer to the go tool.

Problem is that I cannot force a developer to build their Go projects with

gb, so that I can then assume I can use gb to build it later. The most likely path is that a developer will use the official go tool and structure their project (including the vendor directory layout) to match.

Yeah, but is it fair to say that it doesn't work for you, or them, otherwise we wouldn't be having this conversation?

Then I have to make sure any other developer can clone that project and

build it, regardless of where they clone it. To use gb, I would have to make it a standard that everyone follows.

Gb is only for projects, applications that produce binaries. It has no story for library developers, but in saying that, the cost in terms of perceived comparability, I feel, is not as strong when you approach the problem on a application by application basis.

—

You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17271#issuecomment-256149895, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAcA5FC2Ia4SdHd28Q1Fjazln37huDtks5q3lf1gaJpZM4KJZPO .

justinfx commented 8 years ago

Yeah, but is it fair to say that it doesn't work for you, or them, otherwise we wouldn't be having this conversation?

I wouldn't say gb doesn't work on its own. Its more a case of gb not being able to be suggested as the solution to the problem being proposed here. I was just trying to make a case really for the go tool to adopt some of the concepts of gb, by making the zero value of the go tool more useful.

The most common scenario would be that people are following the official Go project structures and expect projects to be able to be built with the go tool. So with that being said, it would be amazing if the go tool could cope with more situations and try harder to build before failing. Just like gb is only for projects, applications that produce binaries, the go tool could also try to ensure that an application can be built regardless of GOPATH not being set, if it can do so by assuming some defaults.

ironiridis commented 8 years ago

It doesn't seem like these two proposals are in conflict. #17262 proposes a mechanism by which GOPATH can be inferred when the environment variable is missing. This proposes an algorithm for that inferred path. In fact it would seem this proposal actually depends on #17262.

bradfitz commented 8 years ago

Let's see how much of the problem is solved by automatic GOPATH=$HOME/go in Go 1.8 and revisit this in a release or two.

lucsky commented 7 years ago

@bradfitz I really don't see how an automatic GOPATH solves anything, this proposal is about the possible ability to do without the GOPATH.

bradfitz commented 7 years ago

I understand the difference. Hey, listen, I don't want GOPATH either. But I'm also realistic about how fast things move. I'd love to be see it happen sooner than later. Personally I'd like something like git where the "go" command would walk up the tree, looking for a magic directory that's the GOPATH. In Git, that's the .git directory. The Go project avoids dot files for the most part, and we're not going to hide user's code. I'd like to see a magic gopath directory that contains the gopath.

We also need to fix #4719 first and get rid of the $GOPATH/pkg directory entirely, making all build artifacts aggressively cached. That would fix another few open issues and common user confusions too.

Then we you could have structures like:

$HOME/my-projects/fooproj/cmd1
$HOME/my-projects/fooproj/cmd1/main.go
$HOME/my-projects/fooproj/cmd2
$HOME/my-projects/fooproj/cmd2/main.go
$HOME/my-projects/fooproj/gopath
$HOME/my-projects/fooproj/gopath/github.com/you/bar/bar.go
$HOME/my-projects/fooproj/gopath
$HOME/my-projects/fooproj/gopath/vendor/github.com/

And if you were in the "cmd2" directory and ran go install, it would walk up, file "gopath", and prepend that to your $GOPATH environment, which will likely be empty for new users in a GOPATH-less world.

"go get" and everything would work.

"go install" would need a bin directory to write to. $GOBIN exists and would be used first. If unset, it could be write to $GOPATH/bin. (yes, there's no gopatch/src in this proposal. Maybe it could exist, but it seems unnecessarily deep) Or we put the default $GOBIN elsewhere.

Anyway, the point is that I also don't like $GOPATH.

It's just a lot of design & implementation work, and nobody's on it, at least yet.

bradfitz commented 7 years ago

Or, a revised version of the above proposal:

Instead of the magic directory being gopath, instead make the magic directory be named src. When walking up the filesystem to the root, whenever /foo/bar/src is found, /foo/bar is prepended as a $GOPATH.

Then the implicit $GOPATH/bin would be $YourProjectRoot/bin, next to your src directory. Then you'd just .gitignore the bin directory, as is typical already.

So:

$HOME/my-projects/.gitignore          # with bin
$HOME/my-projects/fooproj/cmd1
$HOME/my-projects/fooproj/cmd1/main.go
$HOME/my-projects/fooproj/cmd2
$HOME/my-projects/fooproj/cmd2/main.go
$HOME/my-projects/fooproj/src
$HOME/my-projects/fooproj/src/github.com/you/bar/bar.go
$HOME/my-projects/fooproj/src
$HOME/my-projects/fooproj/bin/cmd1   # binary from 'go install' in the cmd1 dir
$HOME/my-projects/fooproj/bin/cmd2   # binary from 'go install' in the cmd2 dir

And there would never be a pkg directory, because we'd do #4719 as a dependency, which would replace pkg with a build artifact cache directory.

kardianos commented 7 years ago

@bradfitz I think using src as the magic would be work. I would propose something similar, but slightly different. We specify the root folder's virtual location with somehow*. We still install bin (and until pkg get's sorted out, pkg too) locally in this root.

$HOME/my-projects/fiiproj/special-file.json < "specifies github.com/you/bar"
$HOME/my-projects/fiiproj/cmd/cmd1/main.go < "imports github.com/you/bar/bears"
$HOME/my-projects/fiiproj/bears/black.go
$HOME/my-projects/fiiproj/bin/cmd1
$HOME/my-projects/fiiproj/pkg/... < until this can go away, have it live here
$HOME/my-projects/fiiproj/vendor < optional can still have vendor folder

This can live in tandem with existing $GOPATH. Idea from here: https://docs.google.com/document/d/1GiSDQHFo5YfYMN5TBaYdjfWHI3KDyfaU1kTHzcLR2qE

In Java / C#, I also often expect a local "bin" and "obj" or similar directory, so I wouldn't make #4719 a dependency, just when that gets resolved, the pkg dir would no longer be needed. The local "bin" would stay.

A dependency of this would probably be a standard "go dep" tool to put the root path specifier.

A small side note, govendor and godep write down the path in their manifest files today so services such as heroku can build the project with just the vcs repo.

mcluseau commented 7 years ago

@bradfitz #14566 is not a duplicate of this... it's the case where we (may) have a GOPATH but want to build something from outside of it. This issue won't solve it at all. To solve #14566, we need to be able to specify the package we are in (for instance with a "--local-package=foo/bar" flag) and have everything else work by default (including the vendor directory).

bradfitz commented 7 years ago

It might not be an exact duplicate, but the issues are all closely related. Using this as the tracking issue is good enough.

mcluseau commented 7 years ago

Ok so now this issue also includes solving the need to override the local package's "reference" when outside the GOPATH. From the top of my head, 2 proposals where made: the flag solution I just put and the "godoc" like comment solution (ie: package quota // import "k8s.io/kubernetes/pkg/quota").

sanmai commented 7 years ago

Don't see anyone suggesting this, but what if we get away from all magic altogether?

Consider there is ~/.goconf in which any developer defines his preferences, plus a per-project .goconf for settings specific to a project. Go tools are expected to look for this file not unlike Git looks for .git/config, .gitconfig and /etc/gitconfig in succession.

This approach gives great flexibility and great transparency for all. There come shorter onboarding times for new developers. There comes understanding. There come things people are craving for. But hatred or heated discussions, endless issues or proposals? Not anymore! Things of past.

No more looking for multiples of $GOPATH set in multitude of dot files. No more questions without answers. Just type go config and here's it all before your eyes.

Even if there's such file, it doesn't mean that there aren't sensible defaults, be it $HOME/go or else. In other words, what was working will continue to work.

(These files are not meant for dependency management. Current approach with vendor already comes with needed flexibility and transparency.)

davecheney commented 7 years ago

I've made several experimental tools along these lines.

github.com/constabulary/kang github.com/constabulary/kodos

On Thu, 5 Jan 2017, 12:39 Alexey Kopytko notifications@github.com wrote:

Don't see anyone suggesting this, but what if we get away from all magic altogether?

Consider there is ~/.goconf in which any developer defines his preferences, plus a per-project .goconf for settings specific to a project. Go tools are expected to look for this file not unlike Git looks for .git, .gitconfig and /etc/gitconfig in succession.

This approach gives great flexibility and great transparency for all. There come shorter onboarding times for new developers. There comes understanding. There come things people are craving for. But hatred or heated discussions, endless issues or proposals? Not anymore! Things of past.

Even if there's such file, it doesn't mean that there aren't sensible defaults, be it $HOME/go or else. In other words, what was working will continue to work.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17271#issuecomment-270540570, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAcA1KhSSIm095QxZeHfUTiSvmMctcuks5rPEnhgaJpZM4KJZPO .

mcluseau commented 7 years ago

@sanmai I'm in favor of anything that can make my use case work ;) the dot-file is another solution.

ascotan commented 7 years ago

O.K. I'm going to bring the now defunct #14566 into this because it has been requested that all discussions regarding that proposal be continued here.

Use cases for #17271 (this proposal):

(case 1) user checks out a go project, hits 'go build' and it magically builds without having a set GOPATH environment variable because the dependencies are in the ./vendor folder.
(case 2) user checks out a go project, hits 'go get' on a dependency and it magically installs it into the local ./vendor directory with no GOPATH set

This proposal does not address what happens when a user calls 'go install' with no GOPATH set. (probably a large oversight)

Use cases for #14566 (locked/dead proposal):

(case 3) user checks out a go project, hits 'go build' and it magically builds WITH having a set GOPATH environment variable. This works because a local ./vendor directory overrides lookups in the configured GOPATH

Proposal #14566 doesn't say anything about the functionality of go get and how that works.

For my purposes I have a GOPATH, but when I use a tool like glide outside of the GOPATH it doesn't work because of the requirement that the entire project must reside in the GOPATH.

The similarity between these proposals is in that the code can live outside of GOPATH and work. Proposal #17262 doesn't change the fact that these are still a valid proposals. There is still a desire to run the go toolchain on code that lives outside of the GOPATH.

In my mind, #17262 actually solves some of the issues with this proposal because it defines where pkg/ and bin/ would show up if you ran 'go install' on code outside of the GOPATH.

Personally I think that if you add the ability to:

Run the toolchain on code outside of the GOPATH where it defaults to $home/GO if it's not defined
Allow local vendor/ folder to take precedence over installed dependencies in GOPATH

this move the ball forward.

The issue that this proposal is proposing that #14566 was not, is that 'go get' should be modified to install into a local ./vendor directory. The problem is that even with a $home/GO, if you install a dependency it ends up in $home/GO not in ./vendor and when you go check in the code, the dependency is missing.

There are a number of vendoring tools (like glide) that currently solve this problem already outside of the toolchain. However a 'phase 2 approach' might be able to get 'go get' to install either 'local' or 'global' (cough npm), so that you can toolchain install into the local ./vendor folder.

skelterjohn commented 7 years ago

@bradfitz have you seen https://github.com/skelterjohn/wgo ? I believe it does very close to what you suggest, with some extra stuff on top that you probably don't care about for revision pinning. I mean, I'm sure you care, but I'm also sure it's a different issue.

That is, it discovers the go workspace in the same way that git finds .git. It has worked well, in my experience, and even has a few users that aren't me.

lloeki commented 7 years ago

I am personally using a cd hook that automatically sets (and unsets) both GOPATH and PATH according to the presence of a .gopath file or directory at or above the working directory:

if it's a file and it's empty the containing directory is the GOPATH
if it's a file and it's not empty the content is the path to the GOPATH
if it's a directory, it is the GOPATH

With this in place I can cd into various projects and have things work the way I want by being creative in the way I place the .gopath items for each one (or group of ones) without ever having to set a GOPATH by hand. When it's a "self-contained" project (.gopath directory at the root of the project) I do have to symlink the root of the project at the proper namespace in .gopath/src (as extracted from git remote get-url).

EDIT: my point is, the really hacky part is that symlink, but otherwise that construct allows for me to have things work out of the box with what I find to be a useful convention with a file that can be included into a repo without extra tools nor requirements or assumptions for that repo (such as with wgo: "github repositories that are made to work with go get do not work as wgo workspaces").

sanmai commented 7 years ago

Probably you have seen it already, but anyway... https://github.com/cloudflare/hellogopher

"just clone and make" any Go project

It would be much better if we wouldn't have to resort to makefiles.

FiloSottile commented 7 years ago

https://github.com/cloudflare/hellogopher

_o/ I wrote hellogopher above and meant to comment here about it.

It's not the goal of hellogopher to be a definitive solution, but to show that the user flow is desired and useful, and to act as a stopgap.

I was very happy to find this proposal. I think what's missing from @rasky's is a way to define the import path "cutoff point", which in hellogopher is a Makefile with IMPORT_PATH defined, while here could be a .IMPORT_PATH file, maybe? I'm not as enthusiast about @bradfitz's because it sounds like exactly setting GOPATH to the project root (while standard and more convenient), which forces a user to choose between the "go get" style and this.

mvdan commented 7 years ago

while here could be a .IMPORT_PATH file, maybe?

If dot files are to be avoided, perhaps a magic header similar to +build in a determined Go file like doc.go would be a better option. That would also mean that an extra file could be avoided.

skelterjohn commented 7 years ago

One solution that I've built into the Google container builder service is to use an import comment (ie package main // import "github.com/you/that" to determine things.

On Jan 23, 2017 5:18 AM, "Filippo Valsorda" notifications@github.com wrote:

https://github.com/cloudflare/hellogopher

_o/ I wrote hellogopher above and meant to comment here about it.

It's not the goal of hellogopher to be a definitive solution, but to show that the user flow is desired and useful, and to act as a stopgap.

I was very happy to find this proposal. I think what's missing from @rasky https://github.com/rasky's is a way to define the import path "cutoff point", which in hellogopher is a Makefile with IMPORT_PATH defined, while here could be a .IMPORT_PATH file, maybe? I'm not as enthusiast about @bradfitz https://github.com/bradfitz's because it sounds like exactly setting GOPATH to the project root (while standard and more convenient), which forces a user to choose between the "go get" style and this.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17271#issuecomment-274448867, or mute the thread https://github.com/notifications/unsubscribe-auth/AAUZ1rX4H2BQiU6NA43J-PNe46vpIJ6Tks5rVH6KgaJpZM4KJZPO .

sdboyer commented 7 years ago

Chiming in here quickly, without having really read the details (sorry!), to note that I think there's a very sane path to getting rid of GOPATH - or at least making it not horrible - that's intertwined with the experimental dep tool. We haven't explicitly put it in a roadmap or anything - things are still a bit of a whirlwind - but it makes sense to treat these things together.

davecheney commented 7 years ago

What a tantalising missive. I would love it if you could share some details.

On Fri, 27 Jan 2017, 07:57 sam boyer notifications@github.com wrote:

Chiming in here quickly, without having really read the details, to note that I think there's a very sane path to getting rid of GOPATH - or at least making it not horrible - that's intertwined with the experimental dep tool. We haven't explicitly put it in a roadmap or anything - things are still a bit of a whirlwind - but it makes sense to treat these things together.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17271#issuecomment-275511212, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAcA_qwG5-UrEH6wvYPz6zlBUjKt46lks5rWQisgaJpZM4KJZPO .

sdboyer commented 7 years ago

I know, I'm sorry, I'm really not trying to be coy or withhold information. I almost didn't say anything, but that seemed worse than dropping a ton of info out of the blue later.

It might take a little while to get it written up - a couple weeks? - as it's fairly involved.

sdboyer commented 7 years ago

I finished the writeup a few weeks ago, but was keeping mum about it because it...well, ran quite far ahead. But the cat's out the bag now, so here it is. Please keep in mind the caveat at the top - it's very much a sketch!

jbarefoot commented 7 years ago

There are a number of vendoring tools (like glide) that currently solve this problem already outside of the toolchain.

That depends on what "this problem" is. One problem that is not solved by glide is that the Go compiler will resolve packages from your GOPATH when that's not what you want--you want dependencies to only come from /vendor, because those are where the pinned versions are placed. In other words, if you are working on a dependent package (or adding a new one), and have it in your GOPATH but forget to add/update it in glide (or whatever dependency management tool you are using), your local build will succeed but the build will fail for everyone else using glide. I have found no way to locally, reliably reproduce the build that our build server runs except by cloning a repo to a new root and set the GOPATH such that the repo is the only one in the GOPATH. If I'm missing something here, I'm all ears.

ascotan commented 7 years ago

Yea, I noted here: https://github.com/golang/go/issues/14566#issuecomment-250507888 some hacky script I was using to do something like this. It 'sorta' works? Basically I rm -f the '$gopath/src/xxx' packages and ln -s what's in /vendor when a make target is run. So the next guy that checks this out and runs make, it 'works'. This falls apart when people start doing nested package names like bob/myprogram/mycuteversionscheme however.

DanielHeath commented 7 years ago

A suggestion for those trying to help in-house developers checkout your code somewhere random and build it:

I create a .env file in the root of the project containing the following

export GOPATH="$(pwd):$(pwd)/vendor"
export PATH="$(pwd)/bin:$PATH"

When working on the project, run source .env and everything will work from that point on.

davecheney commented 7 years ago

What happens if they are in a subdirectory and run

source ../../.env

?

On Tue, 30 May 2017, 12:11 Daniel Heath notifications@github.com wrote:

A suggestion for those trying to help in-house developers checkout your code somewhere random and build it:

I create a .env file in the root of the project containing the following

export GOPATH="$(pwd):$(pwd)/vendor" export PATH="$(pwd)/bin:$PATH"

When working on the project, run source .env and everything will work from that point on.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17271#issuecomment-304757693, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAcAza25eP6e3D9fU-HasaBjH8CUR6Nks5r-3qlgaJpZM4KJZPO .

DanielHeath commented 7 years ago

Nothing works in that case, but in practice I'm almost always dealing with one of:

Doesn't understand, follows the instructions exactly
Understands what's going on and doesn't need help

davecheney commented 7 years ago

Sounds like both cases there shouldn't be much trouble setting GOPATH then.

On Tue, May 30, 2017 at 1:30 PM, Daniel Heath notifications@github.com wrote:

Nothing works in that case, but in practice I'm almost always dealing with one of:

Doesn't understand, follows the instructions exactly

Understands what's going on and doesn't need help

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17271#issuecomment-304766596, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAcA9fruwm8yLVGYP3wXE7dTRmFIK8Iks5r-41YgaJpZM4KJZPO .

DanielHeath commented 7 years ago

The first group need guidance when dealing with multiple go codebases which each have their own dependencies.

I've presented a straightforward (if simplistic) way for each project to have its own GOPATH to avoid dependencies 'leaking'.

If you want something more featureful, I'd recommend gb (thanks, Dave).

lloeki commented 7 years ago

What happens if they are in a subdirectory and run

source ../../.env

That's why I made such a thing automatic when cd'ing in (and out, which unsets GOPATH and unshifts PATH). Also, I did not use a generic sourceable file but one with clear, static semantic as I certainly don't want to clone some random repo, cd into it and have possibly dangerous/adversary shell code run.

EDIT: not being shell means it's also handleable by IDEs and editors without needing a bash dependency.

FiloSottile commented 7 years ago

Those are all valid solutions, and indeed before the vendor folder it's what we did at Cloudflare, but:

A) they are all ad hoc, while Go is the kingdom of convention B) they break go get semantics

Hellogopher tries to improve on those issues, but the Go toolchain will eventually have to provide a final answer.

dlsniper commented 7 years ago

B) they break go get semantics

Maybe we should approach this from another angle. Instead of supporting the currentgo get semantics, maybe we can also change how it works in order to accommodate for the world of no GOPATH.

In theory such a place would look like this:

no more GOPATH, implicit or explicit
every project is rooted where the current working directory is
go get will get dependencies in vendor/ only

For example:

project $PWD is under $HOME/projects/cool_project
project depends on github.com/dlsniper/demolib
the user runs go get github.com/dlsniper/demolib in $PWD
a new folder is created in $PWD/vendor/github.com/dlsniper/demolib and the dependency sources are placed there
if demolib has any other dependencies they should also be downloaded into $PWD/vendor

This would be the first step towards having a concrete package manager as well, as then go get could learn about versioning for these packages or how to retrieve all of the different package and versions required by a project (and yes, I'm actually suggesting discarding, or better said, relocating some of the the work put into golang/dep).

As such, this approach would have many other benefits:

we could have something done quicker, as we don't need full support for everything from the very next release, so we have a smaller surface to cover / test / make sure things work
support for getting a specific package version could be added later on
go get would still be useful in the context of a project based workflow
the metadata files needed to fetch the dependencies could be added as a read-only operation to begin with and allow another tool to be created and deal with the complexities of licenses or whatever golang/dep does these days
would not pollute $GOPATH/pkg with source code (something that golang/dep does today)
golang/dep could still continue to exist and be focused on producing the metadata files needed for go get to bring in stable dependencies
library packages would still be go get-able by older Go versions and new Go versions, while the only thing that really would be different is getting applications.

And we could still have the tool falling back to the old GOPATH behavior either via a env flag that could be deprecated / removed in 3-4 versions as well.

Please take a moment, and if needed a couple of steps back, and think about a world where we could potentially have all of this done by Go 1.10 due to changes needed, even if under an experimental flag for go get (we have experimental env flags for the compiler, right?).

What do you think?

FiloSottile commented 7 years ago

For example:

project $PWD is under $HOME/projects/cool_project

project depends on github.com/dlsniper/demolib

the user runs go get github.com/dlsniper/demolib in $PWD

a new folder is created in $PWD/vendor/github.com/dlsniper/demolib and the dependency sources are placed there

if demolib has any other dependencies they should also be downloaded into $PWD/vendor

That pretty much describes gvt. I don't think the "getting stuff into vendor/" is the side that sorely needs fixing, and anyway it doesn't need to be fixed at the same time.

The go get semantics I refer to are the ones by which main packages still have an import path which all tooling assumes, including using go get to download and install a binary.

What Hellogopher (awkwardly in internals) solves is "I have a project in a random place, all deps are in vendor/, what is GOPATH, I want to build this" while not breaking go get for that project. It also has the nice side effect you mentioned of ignoring the rest of the GOPATH.

When I think about GOPATH-less Go, I think about 3rd party tools or golang/dep to fill vendor/, and the standard toolchain learning to do precisely what Hellogopher does (without make or symlinks, and maybe different conventions).

(Still, I personally agree on golang/dep being over-broad in scope and semantics.)

lloeki commented 7 years ago

In theory such a place would look like this:

no more GOPATH, implicit or explicit every project is rooted where the current working directory is go get will get dependencies in vendor/ only

This does not cater for the very real use case where you need/want a single set of non-vendored dependencies shared among multiple projects.

What happens if the user runs go get from $HOME/projects/cool_project/foo as a first step? after having run it from $HOME/projects/cool_project? Would it detect an existing vendor folder above and use that? Are $GOPATH/{bin,pkg} turned into $HOME/projects/cool_project/{bin,pkg}?

dlsniper commented 7 years ago

This does not cater for the very real use case where you need/want a single set of non-vendored dependencies shared among multiple projects.

I'm specifically talking about the real world where projects are separated. It doesn't have to solve all the problems, it just has to solve enough of them. This also can be addressed later on. The point is that if we try to solve all the problems from the first version and can be iterated to solve more problems in future versions.

What happens if the user runs go get from $HOME/projects/cool_project/foo as a first step?

It downloads the dependencies in $PWD/vendor. Or maybe a future version becomes smarter, much like git traverses the directory tree searching for the .git folder this one can traverse the directory searching for the metadata files.

after having run it from $HOME/projects/cool_project? Would it detect an existing vendor folder above?

The user suffers from not reading the manual, which can be fixed by having the user reading the manual. Or the tool is smart enough (see the previous point about directory traversal) for this not to be a problem anymore.

Are bin and pkg turned into $HOME/projects/cool_project/{bin,pkg}?

Yes, that could be a solution.

Again, I'm not trying to solve all the problems from the release, just enough of them to have a decent (not even good) solution to build on top of without too much disruption of the current workflows. Having go get still work and do things as expected is one of them (and people do expect for go get to fetch things into vendor/ today and their expectation is currently broken).

golang / go

cmd/go: modify the Go toolchain to work without GOPATH #17271