proposal: cmd/go: keep GOPATH mode

rcoreilly commented 4 years ago

This proposal is simply to retain the existing GOPATH functionality, along with the go.mod modules.

The rationale is that GOPATH provides an ideal mode of operation for developing across multiple different repositories at the same time. It is otherwise cumbersome to do so with go.mod, requiring e.g., temporary insertion and removal of replace directives (see discussion in #26640).

A major design priority for Go is simplicity, and GOPATH is simple. It provides a great way for new users to explore lots of different code packages easily, browse, modify, experiment, etc.

There are many different types of Go users, but it would be great to retain support for the "hobbyist" / non-corporate hacker who just wants to do things simply and doesn't care about living on the bleeding edge.

It should not be so hard for the two modes to coexist, as the tooling etc is currently supporting both, and I personally find it most productive to develop in GOPATH mode and then have some Makefile targets that ensure go.mod still works for those who want to use that. And all official releases are done in go.mod.

Thus, it may actually be optimal to have both modes available, instead of trying to complicate go.mod so that it does what GOPATH already does so well.

JAicewizard commented 4 years ago

although I do disagree, the editing repos locally is really nice

dmitshur commented 4 years ago

It should be helpful to identify more precisely and write down what types of workflows are currently easier to achieve in GOPATH mode than in module mode. That would make it possible to investigate if the module mode can be improved so that those workflows do not have more overhead in module mode compared to GOPATH mode.

jayconrod commented 4 years ago

I don't think we should maintain GOPATH indefinitely. Ensuring that tools work in both modes isn't free. It also requires time and effort from project authors to test their code and support users in both modes.

I'd much prefer we improve modules so that this workflow is better supported. Editing and releasing multiple modules has been a problem, and we should figure out a solution to.

As @dmitshur said, it would be helpful to explain your workflow and exactly why it doesn't work well with modules. Please consider adding an experience report.

rcoreilly commented 4 years ago

I think the issues are well documented in #26640 -- here's my summary:

Say you have 3 different repos, each depending on the other: A -> B -> C
And you are in the early development stage, so the API in A & B is potentially changing all the time.
With go.mod, you have to add a replace directive in B and C to use the current live source for A and B, but then for each commit (if others are to be able to use the code) you have to remove those replace directives and do the update to current, and remember to do the commit and push in the proper sequential order (A then B then C), and then re-insert the replace directives before continuing to edit.
This discourages frequent, well-documented commits, and encourages errors due to forgetting any of these steps. If you forget to re-insert the replaces, then you can't figure out why changes in A or B are not showing up in C, etc. It is frustrating.

So, while this is not the absolute worst situation in the world, it is a lot of seemingly unnecessary overhead, given the pure transparent simplicity of GOPATH.

With GOPATH, all core developers could just use that mode, and either work in a branch and do nothing at all at commits, or require users to check out tagged releases to get reliable behavior (or use GOPATH when using the latest HEAD), or have a simple Makefile target that updates go.mod to latest revision.

In short: there are different modes of working, one where you're relying on stable existing package API's and want reliability and stability, and another where everything is changing and the "live source" model makes more sense, so supporting both modes separately may make more sense than trying to find one solution that works for both, when each has very different fundamental requirements.

chewxy commented 4 years ago

I would like to chime in with a real life example:

Context

I develop a family of machine learning and deep learning libraries - gorgonia. Roughly speaking the libraries function as such:

gorgonia.org/tensor - Generic multidimensional array
gorgonia.org/gorgonia - symbolic version of (i.e. AST of the expressions in) gorgonia.org/tensor
gorgonia.org/cu - CUDA.

These libraries each do similar but different things. For example, a MatMul in tensor performs the matrix multiplication. In gorgonia it may perform a matrix multiplication, or it may generate a symbolic representation of the matrix multiplication. In cu, the matrix multiplication is done in the graphics card.

Since they do similar but different things, it would be useful to have similar APIs in the packages. The versions are kept in sync, so all the packages have similar APIs at any given version (well, that's the goal).

The dependencies look like this:

gorgonia -> tensor
         -> cu

Despite gorgonia being dependent on tensor, it's often gorgonia that determines the APIs of tensor. This is done, factoring developer userfriendliness.

The Problem

Currently with Go modules, I have found it quite difficult to simultaneously work on all three libraries at the same time. My solution thus far has been to just use GO111MODULE=off.

Now I think Go modules are one of the more brilliant things about Go. However, it isn't very clear how to simultaneously work on newer versions of the libraries.

Other methods I have tried:

Work using Go modules. Say I am working on A and B at the same time, and B depends on A. I first add the APIs to A, publish, then add the latest version as a dependency of B.

This approach works but generates a lot of superfluous version numbers. This is especially true when working to discover the best most userfriendly APIs. Often it's the API of B that determines the API of A, despite the inversion of library dependencies.

Conclusion

It's difficult to work simultaneously on multiple co-dependent library (co-dependent in the abstract sense - the actual dependencies are unidirectional). GOPATH helps with that.

I view GOPATH as the "working bench" before publishing

kortschak commented 4 years ago

It seems to me that the local go.mod idea described in #26640 would address this.

bcmills commented 4 years ago

This proposal does not seem actionable: it is proposing that we not-do something (removing support for GOPATH) which we have not yet proposed to do anyway (#4719 notwithstanding).

Moreover, I think the emphasis on #26640 is telling: note that that issue is still open. It seems premature to prescribe a course of action based on this assumption that that issue, or its underlying use-case, will not be addressed.

(But #26640 itself seems like a bit of a red herring: the real issue to focus on here is the use-case, not the mechanism. #26640 describes one possible mechanism, but I believe the underlying use-case is #27542, and it may turn out that we can better address it by some other mechanism.)

rcoreilly commented 4 years ago

Wouldn't it be good to have a clear plan going forward? If there is a clear plan to retain GOPATH, then it seems like that would change the way that these other issues are addressed?

According to the oft-cited principle of feature-orthogonality, wouldn't it make sense to not try to shoe-horn GOPATH into go.mod, as in #27542 and #26640, and instead focus on doing whatever might be necessary to make the two modes of operation (GOPATH and go.mod) more cleanly supported and well-documented etc (e.g., the GO111MODULE env variable is likely a suboptimal way of switching between modes -- perhaps a command-line switch instead or at least in addition? And at least a better name for the env variable?)

From this perspective, the discussion in #27542 seems to support this approach, as it was largely focused on essentially just replicating GOPATH under go.mod. Again, the proposal here is to just embrace these two modes as serving largely different use-cases, and adopt the principle of orthogonality, etc. Also, again repeating myself, GOPATH is really really simple, and that is a major advantage over trying to replicate its functionality in some more complicated way on top of go.mod. Let go.mod do what it does best, and GOPATH do what it does best, and forget about all the replace directives etc.

bitfield commented 4 years ago

I think this would be a big mistake. Go is often criticised for having more than one way to do certain things (such as declaring variables), and I take that point. For beginners, it's much less confusing if there's one straightforward, standard, official way to do X.

Once upon a time, the standard way was GOPATH. Now it's modules. Don't look back, look forward. If a few people decide to stop using Go because they loved GOPATH so much, I personally can live with that.

komuw commented 4 years ago

Users have expressed directly to the Go team, more than once[1][2], that they are unhappy with GOPATH. Go modules were put in place, in part, to solve that 'unhappiness'.
I do not think we should be keeping GOPATH mode once modules are deemed stable & complete.
If there's any deficiency in modules(that is currently been met by GOPATH); we should fix that rather than keeping GOPATH.

reference:

ohir commented 4 years ago

@komuw

Users have expressed directly to the Go team, more than once that they are unhappy with GOPATH.

Oh! An astounding one percent of users were unhappy. I would like to see how many users with over a year of Go experience who used Go at work complained about GOPATH. If there was more than a single such person, ofc.

Many expressed a need for better dependency management tools. But not one I know had wished for GOPATH to be fierce slayed.

rsc commented 3 years ago

GOPATH cannot provide proper versioning. It cannot provide reproducible builds. It cannot provide reliable downloads through a proxy. These alone are reasons to retire it. Clearly, we need to address the "interdependent module edits" issue (#27542), but we can do that in modules. We don't need to keep all of GOPATH to do it.

GOPATH was nice and seemed simple. And it is, for a single engineer on a single machine. Its complexities are all around being able to build the same thing on a different machine and get anything like the same binary.

If, as I often say, software engineering is what happens to programming when you add time and other programmers, then GOPATH is for programming, not for software engineering. It does not cope with time (versions) nor with other programmers. And Go is for software engineering, not just programming.

Having two incompatible ways to do something - both GOPATH and Go modules - is not the Go way of doing things. We focus on providing one right answer, not two incompatible ones. It's been a rough transition, and maybe we should have sped it up, but it is coming to an end.

GOPATH is holding back the ecosystem and the toolchain. It's time to retire it. Again, we will work to address specific issues in modules, particularly inter-module editing. And if you have other specific concerns, we're happy to listen. But GOPATH needs to go.

ohir commented 3 years ago

GOPATH was nice and seemed simple. And it is, for a single engineer on a single machine.

And still it is used (is the only straightforward way) to experiment with, or debug interconnected modules. Please come with a solution to the #26640 issue first, then GOPATH may retire.

navytux commented 3 years ago

@rsc says:

Clearly, we need to address the "interdependent module edits" issue (#27542), ... GOPATH is holding back the ecosystem and the toolchain. It's time to retire it. Again, we will work to address specific issues in modules, particularly inter-module editing. And if you have other specific concerns, we're happy to listen. But GOPATH needs to go.

So I second @ohir: to me the right sequence of events is:

1) first fix modules to support "interdependent module edits", 2) then retire GOPATH after "1" is solved.

I recently migrated my projects to modules, but, surprisingly, have to keep using GOPATH for development primarily due to workflow issues when one needs to work on several modules at the same time.

rcoreilly commented 3 years ago

@rsc:

If, as I often say, software engineering is what happens to programming when you add time and other programmers, then GOPATH is for programming, not for software engineering. It does not cope with time (versions) nor with other programmers. And Go is for software engineering, not just programming.

How can you possibly argue with that!

Except, if you transport yourself back to your first experiences in programming, before it got weighed down with all those other people and all that time, and can remember the pure joy of just doing something direct and simple, like writing BASIC on the console of a PC (for me it was a TRS-80 Color Computer, where I tried and failed to write my first AI program in high school, but settled for lots of fun graphics instead :)

I'm sure nostalgia has no standing in the pragmatics of the matter, but anyway if it is possible to make it as easy as possible for people who aren't software engineers to also use Go for the pure joy of it, that would seem to be a good thing.

And there is nothing about modules necessarily preventing that -- just need to figure out these issues.. And speaking of which, I'm having a hard time tracking the current status of things -- it seems like #34506 was implemented but it doesn't solve the inter-dependent issues because it only applies to one module? And if you set the GOFLAGS env it applies to all modules? Is someone in the know going to update #26640 with a summary and proposal or something? Seems like it is in some kind of limbo state?

rsc commented 3 years ago

Based on the discussion above, this proposal seems like a likely decline. — rsc for the proposal review group

purpleidea commented 3 years ago

GOPATH is incredibly useful for everyone who doesn't like go.mod. A very simple reason why go.mod is a mistake: it duplicates functionality that can already be had with git submodules. Golang already depends on git for imports, but then they decided to not be internally consistent and they built go.mod.

The git tools for working with different library versions with submodules work great and it makes it easy to allow most deps that are stable to be constantly built from their git master versions, instead of requiring them to be specified in go.mod.

rsc commented 3 years ago

People do use source control systems other than Git with Go.

rsc commented 3 years ago

No change in consensus, so declined. — rsc for the proposal review group

purpleidea commented 3 years ago

@rsc

People do use source control systems other than Git with Go.

I hope that you'll take a quick moment to reconsider your comment. I feel you might have just pre-made up your mind, and so I hope you'll consider this:

Of course people use other systems than git, and you can use a submodule equivalent in those systems too! Just as a single golang project doesn't require every dependency to be in git, doesn't mean this can't continue working the way it was.

Firstly: most people are on git anyways Secondly: even if I personally don't want to use go.mod, this proposal is about keeping GOPATH and not breaking this perfectly good and working experience for everyone who's using it!

I realize most Google people don't care about this because they have their own internal tools that do all of this for them, but hopefully you'll not break GOPATH for everyone that has been using it since the very beginning.

Thanks!

antichris commented 3 years ago

This addresses an older https://github.com/golang/go/issues/37755#issuecomment-596792488 but since solutions for some of the issues raised therein are older (by an order of magnitude) than that, I felt this may still be acutely relevant to at least some.

@rcoreilly

Say you have 3 different repos, each depending on the other: A -> B -> C

And you are in the early development stage, so the API in A & B is potentially changing all the time.

That dependency graph (A -> B -> C — "A depends on B, which in turn depends on C") seems to contradict the subsequent statement and the rest of the post, so for the purposes of the discussion I'm assuming the other way around (C -> B -> A, "C depends on B, which depends on A") was intended.

With go.mod, you have to add a replace directive in B and C to use the current live source for A and B, but then for each commit (if others are to be able to use the code) you have to remove those replace directives and do the update to current, and remember to do the commit ... and then re-insert the replace directives before continuing to edit.

This discourages frequent, well-documented commits, and encourages errors due to forgetting any of these steps. If you forget to re-insert the replaces, then you can't figure out why changes in A or B are not showing up in C, etc.

Pick.

git add -i / git add -p — these flags have been there since at least 2009 and many GUI tools (including the boring built-in git-gui) provide similar functionality. Use them to pick which changes should go together into a single commit. Always pick changes that go into a commit, not just plop a bulk commit of everything that has changed. I can't stress this point enough, it has been the pitfall of many a budding developer.
Stash.

git stash [push] (blogged about since 2008) can be used to hide away the changes one is not ready to commit just yet, but that are in the way (of e.g. rebase/merge/pull) at that particular moment. A git stash -p invocation, just like the above, lets one pick which of the changes are going to be stashed versus which are kept around uncommitted.

Run git stash pop (or git stash apply) to "unstash" changes. This may cause a conflict if the stashed lines have been altered in the meantime, but such conflict does not require a commit to be marked as resolved: simply edit (or use an interactive merge tool of your choice) to pick and retain the desired changes to be committed when/if you feel ready at a later point.
Assume unchanged.

git update-index --[no-]assume-unchanged (also mentioned by https://github.com/golang/go/issues/26640#issuecomment-756133736) instructs Git to simply overlook a changed file, although this approach may not fit all workflows since operations that change the repository index may reset this option and, in the case of go.mod, you may need some of it committed every once in a while after all.

Here is yet another point where the design brilliancy of Go modules shine: since go.mod is maintained by an automated tool, you can rely on it having a consistent format, one in which the replace directives are conveniently set together and apart from the rest of the content — perfect for excluding when staging changes for a commit.

... and push in the proper sequential order (A then B then C), ...

But of course you have to publish the changed dependencies before you can publish whatever depends on the changes! That just is the reality of software development, regardless of languages, libraries or dependency management systems; that's the order in which you also have to publish the changes when using GOPATH.

This is what I felt I had to vent over complaints about Git workflow with Go modules and ISTM this also pretty much addresses the rest of the points in https://github.com/golang/go/issues/37755#issuecomment-596792488.

Concerning Go modules versus GOPATH, I believe https://github.com/golang/go/issues/37755#issuecomment-771879911 puts it more eloquently than I ever could. I highly recommend reading the excellent series of articles on the design of versioning in Go by Russ Cox, it greatly helped me understanding the design decisions I initially felt were "funky"/"unorthodox" but actually make perfect sense in the big picture.

rsc commented 3 years ago

I realize most Google people don't care about this because they have their own internal tools that do all of this for them,

For what it's worth, on a day-to-day basis I only use the open source Go tools. I go months at a time without using any of Google's internal build tools or source control, and the same is true for most of us working directly on the open-source project. So the assertion that the decision to retire GOPATH is overfitting to some Google-internal development system is untrue.

but hopefully you'll not break GOPATH for everyone that has been using it since the very beginning.

As @antichris linked above, I hope the rationale in https://github.com/golang/go/issues/37755#issuecomment-771879911 is pretty clear that GOPATH does not work for many cases. There is also a very high cost to keeping it, because we cannot make it work as well as modules. It complicates every new thing we want to do in the go command, and it greatly increases the support burden. But most important of all it fragments the ecosystem.

As I also noted in that comment, the major thing we need to fix is to provide some way to work on a collection of modules simultaneously during development, even though they will get released separately. We have some ideas that not quite well-formed enough to state clearly, but we intend to do something for Go 1.17 to address this.

ssxcxS45XT commented 3 years ago

@rsc > There is also a very high cost to keeping it, because we cannot make it work as well as modules.

But there is a cost that removing it causes troubles to old code using GOPATH that you didn't count. I don't think your justification make sense in all cases. Therefore, GOPATH is better to be maintained to ensure backward compatibility.

antichris commented 3 years ago

@ssxcxS45XT No. Keeping GOPATH would have propagated the burden of maintenance: this awkward initial design choice would have to be babysat both in tooling and in that "old code". OTOH migrate the latter to Go modules once, and then that pain is over and done with.

ssxcxS45XT commented 3 years ago

@antichris This justification of removing GOPATH is single sided. GOPATH certainly is better than forcing modules in some situations. I don't think the maintenance burden argument makes sense at all.

OTOH migrate the latter to Go modules once, and then that pain is over and done with.

This statement is laughable. You clearly lack a basic understainding of the evoluation of languages and associated toolsets. You never ending pains. It is already causing pains in cases when modules are unnecesarily.

The current changes in Go seems to be like the wrong direction like C++ is taking. It is trying to adding more features to solve problems previously introduced. C++ is alreadying becomnig too verbose to use. If Go goes down in this wrong route, it will cause more problems instead of eliminating them.

jayconrod commented 3 years ago

@ssxcxS45XT Please keep the Go Community Code of Conduct in mind in discussions here. Your comment above is not in the spirit of that.

ssxcxS45XT commented 3 years ago

@jayconrod Please stay on fact that whether the comment of @antichris is single sided or not.

jayconrod commented 3 years ago

@ssxcxS45XT There's nothing wrong with someone making a comment that supports their position as long they do so respectfully.

ssxcxS45XT commented 3 years ago

@jayconrod This is also nothing wrong to point it out that it is biased and lack of a solid foundation, and probably some basic knowledge.

mvdan commented 3 years ago

Calling someone's argument laughable and implying that they lack a basic understanding of a concept is not respectful.

ssxcxS45XT commented 3 years ago

@mvdan Is the original comment of @antichris biased or not? If it is, then it shows the lack of understanding of basic concepts of software engineering. To make a good design decision in software engineering, it requires unbiaseness. Lacking of such a knowledge, yet still trying to justify such an argument with a biased statement is laughable. It is as simple as that.

antichris commented 3 years ago

@ssxcxS45XT

I understand your frustration. I've had my own fair share of dealing with messy projects over my decades of development. But it is based of this experience that I can also tell you that almost everyone who started out with a big sloppy heap of globally shared dependencies have taken steps to migrate to (or at least, implement) some form of isolated, modular dependency management.

When the initiative doesn't come from the language designers and maintainers, it often is the community that takes action to improve the situation. To name a few, Composer, Bower, Bundler, Maven, NPM, NuGet and PIP — none were there when the language first came out, none of them were originally developed and released by the official language maintainers, and a select few of them have been embraced and accepted as the official dependency management tool by the respective language project (if there even is one); it is only Rust with Cargo, who were built from the ground up with each other in mind, AFAIK. But all these tools are indispensable in their respective roles, and engineers nowadays would not imagine achieving their goals without them.

One could even argue that Docker is a tool that serves the same purpose — bundling a piece of software with its dependencies in a reliably reproducible unit of deployment.

The current changes in Go seems to be like the wrong direction like C++ is taking. It is trying to adding more features to solve problems previously introduced. C++ is alreadying becomnig too verbose to use.

Yes! You're getting it! The Go's plan here is precisely to ditch the arbitrary pile of shitty features that is GOPATH altogether and replace that with a sleek, meticulously engineered system, one that can be depended upon by users and tool developers alike. Read this and these, if you haven't yet, you'll walk away a changed person, hopefully for the better.

In my very personal and highly subjective opinion, world (at least, the computing one) would be a better place if everyone adopted Go modules' approach to the software dependency management problem: installing an application you choose the version you want; its dependencies, having been picked and tested as a whole by the developers, do not affect the dependencies of any other application, agree among themselves upon the lowest common acceptable versions, and are not copied to a huge number of instances between various applications.

I remember back in the day when I was still working with NPM projects, we had tens of gigabytes of the same dependencies copied over and over again, each project having a maximum of few tens of megabytes of original content, but gigabytes of copies upon copies (at slightly different versions) of dependencies. And the Composer projects that could not take new dependencies (i.e., you can't add another library or tool) simply because they were stuck in a needle-point-narrow compatibility spot of version constraints; one could spend days trying to debug and patch through the dependency hell. Truly awful shit, I couldn't be more happy I'm done with it.

So, yeah, rest in peace GOPATH, you shitty pile of trash.

rcoreilly commented 3 years ago

@antichris take it easy there John Oliver! :)

I do agree that Go modules are really great (modulo the multi-module issue), but like all things it took a bit of experience to really appreciate how well they work, and once you get comfortable with the new model, it is hard to see going back. That said, I still spend 100% of my time developing in GOPATH mode due to the multi-module issue (and some kind of weird slowdown associated with #29427 that I need to investigate further). Looking forward to the 1.17 plan!

benitogf commented 3 years ago

I felt a lot of friction comming from other languages with package managers. finding this GOPATH thing that we had to set to be able to run the Go hello world was not a good first time user experience.

However with time I learned how using forks as version locks and go get <fork> was the best package manager, not having to deal with package manager issues was a burden off my team, coordinating to pull an updated repo is not that hard after all... which is something that I cannot say not only about gomodules but about any package manager that I have tried.

from: https://blog.golang.org/go116-module-changes We plan to drop support for GOPATH mode in Go 1.17. In other words, Go 1.17 will ignore GO111MODULE. If you have projects that do not build in module-aware mode, now is the time to migrate.

A little copying is better than a little dependency.

GOPATH is holding back the ecosystem and the toolchain. It's time to retire it. Again, we will work to address specific issues in modules, particularly inter-module editing. And if you have other specific concerns, we're happy to listen. But GOPATH needs to go.

Don't panic.

Please consider not breaking something that its not broken for a lot of people, I think that theres a lot of value in that. Thanks for all the efforts, looking forward to 1.17!

complyue commented 3 years ago

GOPATH is for programming, not for software engineering.

I have to say, a single directory GOPATH is for programming, but GOPATH=/globally-cached/go-deps:/personal-tinkering/go-devs:/team-tinkering/go-pkgs with organization-wide shared filesystems, is for software engineering.

go.mod currently lacks cascaded multi-workspace capability in the design, for software engineering, as it appears today.

golang / go