cmd/go: should default to use mod=vendor flag if the vendor folder exists

JeremyLoy commented 6 years ago

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (`go version`)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

GOHOSTARCH="amd64" GOHOSTOS="darwin" GOOS="darwin"

What did you do?

Given a fresh install of go, there are no files downloaded in $GOPATH/pkg/mod.
I cloned a repo that uses the new Go 1.11 modules with vendored dependencies
I executed go run main.go to run the application.

What did you expect to see?

the application ran without downloading external dependencies.

What did you see instead?

The go tool downloaded all dependencies to $GOPATH/pkg/mod, despite the fact that the dependencies were vendored

I know this can easily be fixed by instead go run -mod=vendor main.go, but because the go tool is now being advertised as "context aware", I think this would be a good QoL change.

thepudds commented 6 years ago

@JeremyLoy note that while the default is to ignore the vendor directory, you can relatively easily change that behavior with an environmental setting GOFLAGS=-mod=vendor. You could use that in CI, or could set it in a .bashrc or similar for day-to-day use if you want.

This is a new environmental variable in Go 1.11, where one of the justifications for introducing it was specifically to make it easier to opt in to vendoring given some people would want that regularly (e.g., see #26585 or https://go-review.googlesource.com/c/go/+/126656).

From the doc:

GOFLAGS
    A space-separated list of -flag=value settings to apply
    to go commands by default, when the given flag is known by
    the current command. Flags listed on the command-line
    are applied after this list and therefore override it.

One other quick note is that those downloads end up in a shared cached (so even if you don't opt-in to vendoring, it cuts down on how many times you would end up downloading dependencies).

rasky commented 6 years ago

I strongly agree with this proposal. If a project committed a vendor folder, this is a strong indicator that vendoring should be used by default. I can’t think of a workflow in which a developer would commit a vendor folder but prefer other developers to ignore it by default.

In other words, if a project decides to use vendoring, then the whole go tool should default to use vendor first, and download/cache only if not there.

FiloSottile commented 6 years ago

By the way, this is part of what GOFLAGS was added for: https://go-review.googlesource.com/c/go/+/126656

rasky commented 6 years ago

GOFLAGS isn’t the right solution here. This is not a per-user / per-computer decision like “add debug flags”. This is a per-project decision that should affect all users building it. Currently, if a project uses vendoring, there is no way to let all builders use vendoring by default.

JeremyLoy commented 6 years ago

WhileGOFLAGS is a decent immediate workaround, I do want to stress the importance of the vendor folder as a part of this issue.

To put it bluntly, Github is not an artifact repository. At any given time, a repository owner can either

delete their account
delete a repository
force push deletions to master
remove a tag
change a tag

To avoid any of the above damaging your project, vendoring is a necessity, and should be treated as a first class citizen in the go toolchain.

There seems to be a rush to remove vendor from the go toolchain. It is still referenced as an "experiment" and there was a bit of swirl in even getting modules to support vendoring in its first release. Until Project Athens or equivalent becomes mainstream, vendoring should be fully supported (but never required).

EDIT: coincidentally, I found out the official Heroku build pack for Go uses this very workaround. It seems like this really should be the default case.

myitcv commented 6 years ago

@rasky

Currently, if a project uses vendoring, there is no way to let all builders use vendoring by default.

But we're talking about working in module mode here, correct? Or by "currently" do you mean GOPATH-mode? Because vendoring as a "first class citizen" continues to work in GOPATH mode, it's just that in module mode it is not the default.

@JeremyLoy

To avoid any of the above damaging your project, vendoring is a necessity, and should be treated as a first class citizen in the go toolchain.

There was a similar discussion over at https://twitter.com/_myitcv/status/1033824806930534400. Vendoring (as it was and still is in GOPATH mode) is one way of dealing with this problem. As I suggested, having a separate repo as a module download cache is another, which actually have a number of derivative benefits.

As you say, Athens (or indeed anything else that talks the published module protocol) are also solutions, solutions that in some cases try to solve even bigger problems.

There seems to be a rush to remove vendor from the go toolchain

I don't think this is quite accurate. In GOPATH mode (the default), vendoring works exactly as it has done up until now. In module mode (the opt in) the behaviour changes.

So for the record, in module mode I am against what is being proposed here, not least because of what I wrote above, but also because it confuses/complicates the rule that "vendor directories are totally ignored in module mode" which is particularly important when it to dependencies.

rasky commented 6 years ago

Because vendoring as a "first class citizen" continues to work in GOPATH mode, it's just that in module mode it is not the default.

Yes, I'm speaking about module mode. The point is that "not being the default" for projects that uses vendoring (that is, projects in which developers went through the process of calling go mod vendor and commit the vendor directory) is the wrong default. If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used. People that don't use vendoring don't need to have vendoring explicitly disabled; they simply won't have a vendor directory in their project. By switching the vendoring support off in module mode, go 1.11 is making using vendoring more difficult for no reasons I can understand, without making things simpler for those who don't need or want vendoring.

There was a similar discussion over at https://twitter.com/_myitcv/status/1033824806930534400. Vendoring (as it was and still is in GOPATH mode) is one way of dealing with this problem. As I suggested, having a separate repo as a module download cache is another, which actually have a number of derivative benefits.

Thanks, but that is largely suboptimal. It requires a custom installation step to be repeated for each user, and also handle and commit binary files into the SCM which are harder to handle (especially with git) and review.

As you say, Athens (or indeed anything else that talks the published module protocol) are also solutions, solutions that in some cases try to solve even bigger problems.

I fail to see how Athens solves the needs of people using vendoring. AFAICT Athens helps with the issue that people can remove/sell GitHub repositories to other parties, and this can break builds and/or even cause security problems. This is not what vendoring is after. Vendoring is required in workflows that need 100% of the source code for a project committed under a single SCM, so that it can be fully audited, versioned, and built completely offline, without any dependency on any third-party service or installation. I deliver code to enterprises that want the code to be complete, not depend on third party services, and fully versioned with their existing code tracking tools. Vendoring solves this problem; Athens doesn't seem to.

So for the record, in module mode I am against what is being proposed here, not least because of what I wrote above, but also because it confuses/complicates the rule that "vendor directories are totally ignored in module mode" which is particularly important when it to dependencies.

You seem against adding a feature for people that need it for the sake of orthogonality. If you don't want or need vendoring, there's nothing in vendoring support that creates problems to you. Just use modules, your download cache, Athens or whatever you prefer. You will never have a vendor directory in your repos and nothing will break or be suboptimal for you. But if I need vendoring support and commit a vendor directory, why should vendoring support be disabled by default for me, causing more difficult builds for all involved developers?

bcmills commented 6 years ago

If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used.

That's not at all “obvious” to me. In fact, the current structure (in 1.11 module mode) is exactly the opposite: if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.

Giving their users an option is not at all the same thing as providing an explicit recommendation.

Moreover, vendor directories can only reasonably work at the topmost level. If a developer makes edits within the vendor folder (intentionally or otherwise), they would effectively perform an implicit replace operation without surfacing that fact in the rest of the UI (such as in the output of go list).

rasky commented 6 years ago

If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used.

That's not at all “obvious” to me. In fact, the current structure (in 1.11 module mode) is exactly the opposite: if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.

I'm not sure that we can deduce anything from what Go 1.11. It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures. What we ended up with is basically non working for many vendoring use cases, hence this bug (and others).

Rather than discussing hypothesis made by people that don't vendor, I'd rather somebody that does use vendoring in production shows up and say that they absolutely don't want vendored modules to be used by default unless explicitly opted in (like Go 1.11 does), and explain us why with a small experience report. I for sure know this doesn't work for my use case as already explained in mine, and others have already chimed in.

Moreover, vendor directories can only reasonably work at the topmost level. If a developer makes edits within the vendor folder (intentionally or otherwise), they would effectively perform an implicit replace operation without surfacing that fact in the rest of the UI (such as in the output of go list).

That's perfectly fine for me at least. In fact, in my use case, vendoring support could go as far as completely subsuming the module download cache and/or go.sum. That is, it would be great that once you opt-in to "vendoring mode", all new modules are automatically expanded into the vendor directory, used by default, removed when they're not used, etc.

bcmills commented 6 years ago

I'd rather somebody that does use vendoring in production shows up and say that they absolutely don't want vendored modules to be used by default unless explicitly opted in (like Go 1.11 does), and explain us why with a small experience report.

That's kind of my point, though: you're requesting feedback from one particular subset (package maintainers that choose to use vendoring) and ignoring another (package consumers, who have no choice in the matter in GOPATH mode, but do now have a choice in module mode).

We need experience reports from everyone — including package maintainers and consumers, and including both open-source and enterprise codebases.

bcmills commented 6 years ago

@JeremyLoy, could you give some more information about your use-case?

The go tool downloaded all dependencies to $GOPATH/pkg/mod, despite the fact that the dependencies were vendored

Did you run go mod download after cloning the repository? Why or why not?
What is the impact of downloading the dependencies?
- Do you intentionally want to bypass go.sum verification, or do you have in mind that the vendored dependencies remain unmodified?

Until [module proxies become] mainstream, vendoring should be fully supported (but never required).

Vendoring is still supported in Go 1.11, both in GOPATH mode and with -mod=vendor. I am not aware of any proposal to remove it in 1.12, and would be surprised to see one at this point.

bcmills commented 6 years ago

@rasky

It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures.

Please remember that among our gopher values is to “be charitable”.

The approach to vendoring in Go 1.11 does seem to have evolved with the implementation, but that doesn't mean it was “retrofitted”. A flag very similar to -mod=vendor was floated in #19234, and the ability to disable vendoring was discussed in #16562; as far as I am aware, both of those significantly predate the current module design.

JeremyLoy commented 6 years ago

@bcmills I'm currently using Go for enterprise application development, so I can give an experience report for that. You'll have to find someone else for the package maintainer report.

We currently vendor our dependencies using dep. We looked into converting to go modules, but found that the differences were too great to warrant a change right now, especially as modules are still in development.

We vendor for a variety of reasons, many of which were already stated here, but I will reiterate them for clarity.

a single deliverable - the client is able to take the codebase and build the application from source without internet access.
guaranteed stability - the leftpad problem simply cannot happen with vendoring. It is still an issue with both GOPATH and module development
security audits - the client is able to audit not just our source code, but our dependencies in a reproducible manner.

we also vendor tools such as gomock, goimports, swagger etc.

Now to answer your questions:

Did you run go mod download after cloning the repository? Why or why not?

no. because with the vendor folder, I should not have to.

Do you intentionally want to bypass go.sum verification

no, quite the opposite in fact. I imagined that go.sum would be a drop-in replacement for dep's Gopkg.lock

do you have in mind that the vendored dependencies remain unmodified?

yes.

Having to set an environment variable or constantly add a flag is a bit of a nuisance, but not a game breaker. The biggest issue to me is that I expected go modules + vendoring to behave exactly like dep with vendoring (algorithmic changes not included)

rasky commented 6 years ago

It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures.

Please remember that among our gopher values is to “be charitable”. The approach to vendoring in Go 1.11 does seem to have evolved with the implementation, but that doesn't mean it was “retrofitted”.

I'm sorry if it was borderline harsh, but anything related to "proposals" in Go often leads to a very frustrating experience, so much that I tend to steer away as much as possible from it and focus on where the community experience is vastly superior (eg: compiler development).

This said, I still think the quoted sentence is factually true. It was clearly stated in Russ' first blogpost:

This proposal keeps the best parts of go get, adds reproducible builds, adopts semantic versioning, eliminates vendoring, deprecates GOPATH in favor of a project-based workflow, and provides for a smooth migration from dep and its predecessors.

and has been repeated other times elsewhere. The vgo design didn't include vendoring; at some point, it was re-added to the picture, but there wasn't any discussion or attempt (at least that I could find) to rethink vendoring in the Go modules world. The fact that a default that doesn't work for people using vendoring was chosen, or that there are large overlaps between the vendor directory and the download cache is probably a consequence of not having fitted vendoring in the picture from day one.

thepudds commented 6 years ago

@rasky I think you are right that the initial set of vgo blog posts initially proposed dropping support for vendoring.

However, there was then a fair amount of feedback from the community about the importance of vendoring. As far as I could see, that community feedback was clearly heard by the core Go team, which then resulted in the proposed plan being updated to actually retain support for vendoring with modules. The feedback was discussed in multiple venues, but this might have been the longest thread:

https://groups.google.com/d/msg/golang-dev/FTMScX1fsYk/uEUSjBAHAwAJ

...where within a week or so of the initial vgo blog post, the core Go team had switched to proposing retaining vendoring support with vgo/modules based on that community feedback.

In any event, sorry if my historical comments are not very interesting. A much more interesting question of course is what should the behavior be in 1.12+.

bcmills commented 6 years ago

The vgo design didn't include vendoring; at some point, it was re-added to the picture, but there wasn't any discussion or attempt (at least that I could find) to rethink vendoring in the Go modules world.

Yeah, I can see that. Bear in mind that the module support in Go 1.11 is still preliminary: we know that it has some rough edges, and vendoring is one of them.

I think it's important that we figure out a path forward, but it's not clear to me whether that will be to improve vendoring and better integrate it into module mode, or to improve the available module operations and deprecate vendoring entirely. I expect we'll try something in at least one of those directions for 1.12, but it's important that we have enough time to try out the improvements and make sure they work for those use-cases before they become the default.

In particular, because vendoring support in module mode is pretty awkward in 1.11, I think it's important not to try to push it as a default: I don't think it's the right design point yet, and I don't want folks to start relying on a design that really needs more thought.

ondrej-fabry commented 6 years ago

a single deliverable - the client is able to take the codebase and build the application from source without internet access.

guaranteed stability - the leftpad problem simply cannot happen with vendoring. It is still an issue with both GOPATH and module development

security audits - the client is able to audit not just our source code, but our dependencies in a reproducible manner.

@JeremyLoy has made some great points here why vendoring is really needed in the software we build.

Without vendoring how are you planning to solve these points? Force everyone to host some caching/proxy thingy that would serve the dependencies transparently? This would most likely compel developers to create various different solutions just like with vendor did before (Godeps, govendor..) and thus split the community once again while still not solve all of the points above.

What if some of the dependencies disappears and you don't have it in the proxy cache or anywhere else? This would force developers to fix/replace the dependency in their repository. However what if this is related to multiple versions with ongoing support? It would have to be fixed for all of them..

I'm really curious about your future solutions without the vendoring support that would satisfy the points above while at the same time not force us to manage extra things outside of the repository to successfully build our Go programs.

bcmills commented 6 years ago

@ondrej-fabry

Without vendoring how are you planning to solve these points?

That is a matter for a separate issue (perhaps #27618, and perhaps others as well). If you have specific requirements, please contribute an experience report describing both your use-case and how you address it today. (As I've said, it's not clear to me whether the solution will be to improve vendoring or to replace it, but experience reports will help either way.)

The matter at hand for this issue is whether -mod=vendor should be the default, in module mode, when the module's root directory has a vendor subdirectory.

rasky commented 6 years ago

Is there a single experience report, email or comment on GitHub issue that describes a real-world workflow in which the current default is correct? I have not seen any.

bcmills commented 6 years ago

@rasky I don't think we have any experience reports for vendoring with module mode at all. As I've said repeatedly, I think it needs work before we can recommend any particular workflow. For now, if the module support in 1.11 doesn't work for you, don't use it. GOPATH mode is still supported and shouldn't get in your way.

If you want to influence the future direction of module mode, you need to share your actual use-case instead of just arguing that what we have today doesn't work: your current criticism is not constructive. We already know that what we have today doesn't work for you: we need to know how to make that better.

JeremyLoy commented 6 years ago

@bcmills I added a link to my comments (and by proxy, this issue) to the experience report page you referenced.

If you want me to modify the experience report/submit it in a different way etc just let me know!

driusan commented 6 years ago

My experience with the current vendor behaviour has been as follows:

I've written a (partial) drop in replacement for git in Go, primarily so that I can use it on Plan 9 where there's no access to the real git client. A user requested that I vendor my dependencies so that it can be bootstrapped without @0intro's rc script hack, which seemed like a simple enough enough request so I decided to oblige.

I copied the files to the vendor directory as I did to vendor with previous versions of Go and it didn't work. I ran "go mod vendor" which appeared to generate some ancilliary files for Go 1.11 like vendor/modules.txt. It appeared to work (I was inside my $GOPATH), so I committed and pushed the results. The same user then filed a GitHub issue saying that it was still trying to pull the code that was supposed to be vendored code from bitbucket instead of using the vendored library when doing go build -x. After some investigation, I found the -mod vendor flag and it resolved his problems. (Most of this took place at driusan/dgit#142)

I support this proposal because the current Go 1.11 behaviour:

Isn't how previous versions of Go worked.
Isn't what package maintainers expect.
Isn't what users expect.
Makes it difficult to build offline, even when all dependencies are vendored.

bcmills commented 5 years ago

I think mod=vendor is too coarse: ideally the vendor folder should function more like an implicit set of replace directives, and we should automatically use whatever modules are found there (and fetch the remaining modules as usual).

That's certainly not happening for 1.12, but it's still on my radar.

snabb commented 5 years ago

How about:

Make it possible to specify "vendor" as the module version number string in go.mod. In that case take the module from the "vendor" subdirectory. Otherwise fetch the module as usual.

"go mod init" could add the modules in the "vendor" subdirectory automatically to go.mod with the "vendor" version definition.

That way it would be easy to see from go.mod which module comes from where. No need to look in many places.

nomad-software commented 5 years ago

@snabb Then you lose all version information and we're back to square one.

I'd just like to comment that I'd be totally opposed to removing the vendor folder. For me, it is absolutely necessary to make sure I still have project dependencies that might be taken offline. The vendor support so far in the new module system is ok but I wish it would copy the entire dependency's repository. I've had to write a tool to do this instead. https://github.com/nomad-software/vend

snabb commented 5 years ago

I think people have different use cases for the "vendor" directory.

In my common use case the packages in the "vendor" directory have local fixes, improvements or other customizations which are not available in the original repository. Therefore the vendored module can not have any version number which would correspond with the versioning in the original repository. I just want anything that is there to be used, overriding any other sources and thus the version number becomes irrelevant.

(I do realize that I might be abusing the vendoring mechanism to implement local modifications to dependencies, but it works well for that even if it was not the original planned use case.)

Added to clarify: This use case is in a corporate environment where the source code and the local modifications to the public packages remain private and are stored in a VCS which does not work with Go tools. (Obviously a different work flow would be preferred with public code on Github.)

nomad-software commented 5 years ago

@snabb Take a look at the Replace directive in the mod file. This is exactly what you are describing. Then you could put your replacements in another directory.

snabb commented 5 years ago

@nomad-software Oh, this is great. I did not realize replace can be used for that. The documentation go help go.mod fails to mention it can be used to refer to local file system paths.

yanyandenuonuo commented 5 years ago

I think mod=vendor is too coarse: ideally the vendor folder should function more like an implicit set of replace directives, and we should automatically use whatever modules are found there (and fetch the remaining modules as usual).

That's certainly not happening for 1.12, but it's still on my radar.

@bcmills So the vendor/modules.txt should be commit to VCS?

bcmills commented 5 years ago

Yes, if you're committing the vendor directory you should definitely include vendor/modules.txt.

yanyandenuonuo commented 5 years ago

@bcmills

go dep only create Gopkg.lock, and other language tool such as maven, composer, npm alse only create one file xxx.lock.

So why go mod create two and what's the difference between vendor/modules.txt and go.sum?

thepudds commented 5 years ago

@yanyandenuonuo I am not an expert dep user, but my understanding is that dep uses a manifest file (Gopkg.toml) and creates a lock file (Gopkg.lock) and also populates vendor. E.g., from a list in the dep doc describing the "state" that dep uses:

A manifest - a file describing the current project's dependency requirements. In dep, this is the Gopkg.toml file.

A lock - a file containing a transitively-complete, reproducible description of the dependency graph. In dep, this is the Gopkg.lock file.

The source code of the dependences themselves. In dep's current design, this is the vendor/ directory.

In other words, even with dep, there is a difference between a lock file and vendor.

With Go 1.11 modules, the go.mod files in a build include the dependency requirements as well as provide enough information for 100% reproducible builds (and hence there is not a separate go.lock file or similar in Go modules, and go.sum is not a lock file either).

Go 1.11 modules also provide you with the option of populating vendor. This is not a requirement, but it helps with use cases such as "GitHub is down", a repo disappearing, etc., and it also helps with allowing different projects within the ecosystem to transition to modules at different rates – older versions of Go such as 1.10 understand how to consume a vendor directory created by go mod vendor, so vendoring is one way for a module-based project to provide dependencies to older versions of Go that do not fully understand modules. vendor/modules.txt is an implementation detail of the vendor directory – you normally do not need to examine it and should not manually edit it, but as Bryan said above, you do want to check it in along with the rest of your vendor directory if you are checking in vendor.

These two modules FAQs go into a little more detail, and are probably worth a quick read:

yanyandenuonuo commented 5 years ago

@thepudds Yeah, i had read the FAQS, but there dosn't state more detail about go.sum, and no state about modules.txt. The wiki say go.sum is not a lock file, but it's state just like a lock file. It's seem like a alias name or format of lock file. I think go.sum and modules.txt is repeat in effect.

bcmills commented 5 years ago

@yanyandenuonuo, the topic of this issue is whether the go command should use the vendor directory by default in module mode. See http://golang.org/wiki/Questions for appropriate venues for general questions about modules and vendoring.

yanyandenuonuo commented 5 years ago

@bcmills If vendor/modules.txt is relate with build, so it's ok that discuss the difference between modules.txt and go.sum. If not, i think both of the file shouldn't be commit to vcs.

justinfx commented 5 years ago

FWIW I am in the same boat of maintaining Go projects in a proprietary environment that has limited or no external network access. So I have needed to structure my projects in a self contained way. Before the Go module support, I had to have our custom Go tool for our waf build system handle doing a dynamic GOPATH for the project. We would also recommend using Glide. This allowed for non Go devs to clone to any location and build with confidence from the project with its vendored deps. Since Go 1.11 this has become much more simple when using Go modules + vendoring. But we don't want the build to try and download. So each build has to make sure to set mod=vendor. I am totally onboard with the idea of respecting the project maintainers decision to commit a vendor directory and for that to be the default mode when detected.

but it's not clear to me whether that will be to improve vendoring and better integrate it into module mode, or to improve the available module operations and deprecate vendoring entirely.

I don't see why we would want to remove the option for vendoring when it is clearly a useful approach as suggested by the feedback from vendor users. If a project maintainer wants to use vendoring, then it should be propagated in the same way it is for GOPATH mode. What we have now is inconsistency between GOPATH and module modes.

In particular, because vendoring support in module mode is pretty awkward in 1.11, I think it's important not to try to push it as a default

It is only awkward because the behavior from GOPATH mode to module mode was not consistency retained. If the default mode could be detected as vendor then it is no longer about awkwardly needing to add a build flag. If a project maintainer doesn't want to have vendored deps used, they can remove the directory from the repo. And if a user of that project wants to ensure they always download, they can set the mod flag. Shouldn't we ensure the project owner has options? I see it similar to how a cgo project can choose to set default sane compiler and linker flags for the build, but a user still has the option to manually override those.

cespare commented 5 years ago

I have more feedback from real world experience.

At work we have a monorepo with a single vendor directory. Vendoring everything has been a very stable approach and we're likely to keep doing it for a while, at least until the tooling around modules reaches a similar point of stability.

The main problem we have with our GOPATH-based approach is that the tools we use for managing/updating the vendored dependencies seem to be persistently buggy/hard to use. I looked into replacing these with go mod vendor. My hope was that it would serve a dual purpose of both being more robust and also taking a small step toward in the module direction.

Unfortunately, we would need to use the GOFLAGS hack everywhere (local dev environments, build infrastructure, CI, etc) to ensure that all our go tool commands will use the vendored code. We never want to build the code without using the vendored dependencies.

I also have written/maintained some open-source projects. For library packages, whether or not they've been updated for modules, they never contain vendored code (as has always been best practice), so that's not relevant here.

For applications, either (a) they contain vendored code and expect a GOPATH workflow (no go.mod), or (b) I've updated them for modules and deleted the vendor directory. I have no problem requiring Go 1.11+ for building my applications.

If for some reason I updated an open-source application for modules and used go mod vendor to add a vendor directory, I would want my users to build with that vendored code (same as my work scenario). But I don't anticipate using go mod vendor for my open-source work at all.

I agree with what @JeremyLoy, @rasky, and others in this thread have said. It seems like using vendor for the main module, if it exists, is a much better default.

if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.

I have a hard time seeing this practice coming into widespread use (as to justify the current default). It seems like it would be quite error-prone: wouldn't it be easy for the (Go 1.11+/module-using) developers to alter dependencies in the go.mod file (which various go commands will happily do) and then forget to update the vendor directory?

ondrej-fabry commented 5 years ago

@cespare Please be sure to include your real world experience in experience reports.

cespare commented 5 years ago

@ondrej-fabry my feedback is specific this particular issue; it is not a report on my experience writing Go.

theckman commented 5 years ago

It seems that this issue is still marked as "NeedsDecision". The feedback in this ticket has been overwhelmingly in favor of making this change to be the default. The only dissent I've seen here has come from @bcmills @myitcv. I'm also not seeing anyone indicating agreement with their dissent (either via comment or by virtue of emoji).

@bcmills @rsc what else is needed to make a final decision for what will be better for the entire Go community? How can I help us work towards making a final decision on this issue?

mvdan commented 5 years ago

The issue is already milestoned as 1.13; even if it was implemented today, it wouldn't ship to most users for another six months. Is there a particular urgency to getting this done?

theckman commented 5 years ago

@mvdan everyone adopting modules now who wants to keep vendoring is going to need to add this extra bit to their build scripts, after spending the time trying to figure out why they keep downloading the world, to then remove it later assuming the default is changed. Seems like it could become a pretty large distributed time investment, and would be a bit of churn.

Edit: I've seen it come up on Slack quite a few times, and this ticket has been open since August.

mvdan commented 5 years ago

Sure. But hurrying to approve and implement this issue this week is not going to affect users any faster. I personally think that endorsing vendoring as it exists today isn't a great idea for modules, so I'd like to see what approaches arise during the 1.13 cycle. In my opinion, getting the correct solution is better than getting a solution fast.

this ticket has been open since August.

Note that there are likely hundreds of modules-related issues at any point nowadays; there's tons of stuff that has been pushed back to 1.13, and I'd argue this is not the most urgent issue of them all. This is precisely why GO111MODULE=on is not a default yet.

thepudds commented 5 years ago

As far as I was able to follow, it didn’t seem to me that the core go team was against enabling vendoring by default, but rather there was a bit of a question as to exactly how it should behave when enabled by default, and then of course the time needed to implement.

See this recent comment from @bcmills, for example, which suggests what seems to me to be an improvement to behavior (compared to the current 1.11 mod=vendor behavior):

https://github.com/golang/go/issues/27227#issuecomment-439192517

But of course, different people might have different perspectives as to whether not that is an improvement, which seems like a valid thing to discuss or capture in this issue.

theckman commented 5 years ago

@mvdan I'm not trying to approve and implement this issue this week. I'm looking for a final decision to be made so that the work can start for the Go 1.13 cycle. I'd personally prefer this gets in to 1.12, to short-cut some of the churn, but I don't see that as being a reasonable ask at this point.

I personally think that endorsing vendoring as it exists today isn't a great idea for modules

Can you elaborate further? Why are 100% reproducible builds a bad idea for modules?

I've been on the unlucky end of total loss of an artifact server, and a dependency having been killed in the process. That code was lost and the project could not be compiled. In that case Modules wouldn't have saved me, but vendor would have.

I don't think we should treat vendor as a bunch of replace statements, but instead a way for those who want absolutely reproducible builds to get them. It's my belief it should be a dependency cache that travels with the source repository.

bcmills commented 5 years ago

As far as I can tell we're in agreement that we should use vendor more aggressively by default.

However, there are still some details to decide. For example:

Should we substitute modules, individual packages, or the entire transitive source tree? (Probably modules, but then users will have to be careful to keep modules.txt up-to-date. The vendor directory in the pre-modules world applies to individual packages; on the other hand, -mod=vendor today applies to the entire source tree.)
Should we modify go mod vendor to support more targeted vendoring? (Perhaps, because that may make it easier to maintain an accurate modules.txt.)

FWIW, I'm planning to address this issue in Go 1.13.

nomad-software commented 5 years ago

@bcmills I think we need to first keep in mind what problem the vendor folder actually solves. In my mind, it supplies one hundred percent reproducible builds, which don't need an internet connection and protects you from online repositories being removed. That's probably as sophisticated as we need to get.

bcmills commented 5 years ago

@nomad-software, the vendor folder is not necessary at all for 100% reproducible builds in module mode. You can save the contents of the module cache, and use the saved cache as a GOPROXY. (This was all covered in the preceding discussion.)

The vendor directory does not solve reproducibility; in fact, quite the opposite, since you can make arbitrary changes in the vendor contents that will not be reflected in builds initiated from outside your module. (See #29058 and #27348.)

What the vendor directory does provide is a more ergonomic presentation of the cached code: easier auditing without modifying the repo, simpler diffs for version control tools, etc. Some folks may want that for their entire transitive closure of dependencies, but others likely will not.

bcmills commented 5 years ago

@leitzler points out that using vendor by default would change the source code in use for existing modules that have a vendor directory, which actually works against reproducibility for those modules.

That's true, and one of the details I'm contemplating is “when to activate vendor-by-default”: now that go mod tidy adds a go line to the go.mod file, we could trigger the behavior (or not) based on the declared version. For example, perhaps we should only use the vendor directory by default if the go.mod file specifies go 1.13 or later.

theckman commented 5 years ago

the vendor folder is not necessary at all for 100% reproducible builds in module mode. You can save the contents of the module cache, and use the saved cache as a GOPROXY. (This was all covered in the preceding discussion.)

@bcmills This just sounds like more manual vendoring, where you'll now need additional scripts to overlay the module cache and to resolve / save any new additions. Or you now need to have some sort of process to continually check in your module cache to source control. How do you then use that on different developer workstations and build systems? What tooling will go mod provide to help make that easier?

Thinking about it here it seems complex, with potential for pain in the future. It makes me even more convinced that we should keep the more simple vendor concept as the default.

The vendor directory does not solve reproducibility; in fact, quite the opposite, since you can make arbitrary changes in the vendor contents that will not be reflected in builds initiated from outside your module.

If you assume most people won't be just changing stuff in vendor willy-nilly, it doesn't have the opposite effect. It'd especially be true go mod toolkit were to control that folder and overwrite any changes that are made, and to fail of there were any modifications. I never would advocate for mutating vendor, and if you wanna change something there you need to fork it and have your dependency manager drop the fork in to vendor at the right path. I think replace would handle this nicely.

I also assume that vendor will not be considered for things consuming that module, and that generally the only thing using vendor are tests and the build for package main. We don't run tests from external packages, and we can't import their package main, so that seems fine. And with MVS, new consumers should get the same package versions as in their vendor folder, or the version semantically closest to it.

Now, if one of those vendored dependencies disappears, new consumers will fail to build. But the original project being imported will build on its own. And because it's vendored, it gives new consumers a clean way to copy the dependency in to their vendor folder in their repo and make use of it.

golang / go