cmd/go: provide straightforward way to see non-test dependencies

[rogpeppe]

It's important to be able to see the full set of dependencies used by production code exclusive of tests, particularly with different major versions as distinct modules, as inadvertently using an import with a different major version can introduce bugs (also, some external dependencies can be toxic to have in production code).

The go mod why command provides a -vendor flag to exclude tests; perhaps go list could provide a similar flag to exclude testing dependencies.

In addition to the above, it is also useful to be able to be able to see modules used by local production code and tests, but not transitively by tests in external dependencies, as external dependencies are harder to change, and inadvertently forked modules are still an issue for tests too. I'm not sure of a good way to specify that, though.

[thepudds]

An example where a benchmark-only dependency appearing in go.mod caused consternation: https://github.com/scylladb/go-set/pull/3/files#r208130318

In that example, one of the main points of this new project is to resuscitate a dead project (in a cleaner form), and there was surprise at seeing the dead project listed as a dependency of the new project. (There was surprise and consternation on the part of the authors, but one could also imagine that being a deterrent to a user evaluating whether or not to download and/or use this new project if it appears the new project relies on a dead project).

[flibustenet]

We should see if it's better to have this information directly in the go.mod file (like // test or separate sections of require). Or if it's better to use a cmd or a web site like godoc.org ?

[myitcv]

@flibustenet - just in case you haven't seen https://github.com/golang/go/issues/26913#issuecomment-411976222 yet.

I think it's worth focussing on what people are trying to achieve here... which I agree has some overlap with their workflow and hence the tools they are using, but in the case of using godoc.org, I believe the underlying question that someone is trying to answer is "what dependencies does module M have?" and making some judgement on whether or not to use M using that information. Getting more detail on that will, I think, help to steer us towards the building blocks we then need in place.

As I noted on Slack:

Because all of these sorts of queries are possible using go list.... but the very fact we're having this discussion means that that solution is probably too esoteric. And if the queries are common enough it probably makes sense to provide something more accessible for day-to-day use

That said, we don't need to load everything into the go tool; if it provides the building blocks for some other solution that answers these same questions but via a different UI/UX then that's just as good.

[rogpeppe]

As an example of the issue here, I just noticed a dependency on github.com/jtolds/gls which I wasn't expecting. The go mod why output looked like this:

# github.com/jtolds/gls
github.com/juju/charmstore-client/cmd/charm
github.com/juju/juju/juju/osenv
github.com/juju/juju/juju/osenv.test
github.com/juju/juju/testing
github.com/juju/juju/api/block
github.com/juju/juju/api/block.test
github.com/juju/juju/state
github.com/juju/juju/state.test
github.com/juju/juju/component/all
github.com/juju/juju/cmd/juju/commands
github.com/juju/juju/provider/all
github.com/juju/juju/provider/ec2
gopkg.in/ini.v1
gopkg.in/ini.v1.test
github.com/smartystreets/goconvey/convey
github.com/jtolds/gls

So the shortest path to this dependency takes us through four levels of external test dependencies. In practice I almost never care about test dependencies of external dependencies.

[wsc1]

Hi, I'm redirecting the comment here

Russ's comment about scopes is compelling to have unified go.mod, but not compelling to hide scopes (build tags x test).

I think having go mod why not refer to scopes can be confusing or worse: go mod why is arguably not providing the functionality advertised (i.e. explaining why) if it doesn't also give scope information, at least for the main module.

[dmitris]

I would like to be able to look at go.mod and find out which packages are needed to build and run tests in the current module. In particularly, this is needed if you have a policy that all open-source / "external" (outside of your company control) packages must be mirrored on internal servers.

One idea is for go to add comments like github.com/gopherjs/gopherjs v0.0.0-20181004151105-1babbf986f6f // indirect-test for indirect dependencies that are dependencies from tests in the dependencies' modules. Then one could look at go.mod and see that packages/modules marked as // indirect-test are needed only if you want to run go test all or go test ... but not for the "regular" workflow of go install -v .... && go test -v ./... (to build and test your own code).

[flibustenet]

Tests dependencies should be visible, even for direct test dependencies. It's a valuable information. I'd like to choose a lib with as few dependency as possible to run but the most tests (and test dependencies if needed) as possible...

[bcmills]

In particularly, this is needed if you have a policy that all open-source / "external" (outside of your company control) packages must be mirrored on internal servers.

That's probably a task better suited to GOPROXY than to go.mod. (In particular, I would expect the company to provide a GOPROXY server containing all of the approved/vetted versions of external modules.)

[wsc1]

I would like to be able to look at go.mod and find out which packages are needed to build and run tests in the current module.

I think the go tool would be better. But, in theory the problem of deciding whether or not a file has satisfiable build tags is NP-complete :)

[bcmills]

perhaps go list could provide a similar flag to exclude testing dependencies.

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

[myitcv]

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

Per my answer in https://github.com/golang/go/issues/27900#issuecomment-437926158, I think it depends if you expect/want that answer to be irrespective of build constraints.

[rogpeppe]

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

Yes, this is pretty much exactly what I was after (I wasn't aware of the -deps flag). For the request in my final paragraph, go list -test -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}'| sort -u does the job. Not that either of these are particularly easy to type, but I can write a script. It would be nice if go mod why could work with this call graph instead of the regular one though.

PS I'd be happier if the go command never looked at external test dependencies (they can be enormous and it's often impossible or uneconomic to run all tests in all third party modules), but that's another story.

[advdv]

Story I want to share my frustrating journey to this problem. I recently started a new project and from the get go I wanted dependency management to be done purposefully. Meaning that i want to be careful and conscious about how codes get into my binary.

I'm not building any thing exotic, a basic web app with a database. This is my go.mod file:

module github.com/my/project

go 1.12

require (
    github.com/advanderveer/go-test v1.0.1
    github.com/golang-migrate/migrate/v4 v4.2.4
    github.com/google/wire v0.2.1
    github.com/hashicorp/go-uuid v1.0.1
    github.com/lib/pq v1.0.0
)

As a general policy I wanted all my dependencies to be at version v1 or higher. So unfortunately wire stands out here. I though that wire was only for code generation purposes so I was confused as to why it was here. Ergo, i run go why github.com/google/wire and it gives me no new information as to "why" it is there even though i guarded it from building //+build wireinject flag. Googling around a bit I start to accept that the whole build flag thing is a bit confusing for determining what modules are important here. So i cut my losses and simply add a comment to the mod file for the future, I'm unsure if this comment stays intact why I'm adding more dependencies but I have bigger fish to fry:

Because during my research above I went on to to open my go.sum file. Even though I managed my dependencies carefully I somehow ended up with 200 lines of packages I don't directly seem to use. Some of them quite big:

...
cloud.google.com/go 
github.com/aws/aws-sdk-go
github.com/cockroachdb/cockroach-go
github.com/gogo/protobuf
github.com/gopherjs/gopherjs
google.golang.org/grpc
...

So why are they there? go mod why github.com/gogo/protobuf says: # github.com/gogo/protobuf (main module does not need package github.com/gogo/protobuf). Eehm Ok. So why are they listed there? After some googling I run into this reddit thread which refers me to this issues where it is said: "If you have dependencies only needed for tests, then they will show up in your go.mod, and go get will download their go.mods, but it will not download their code" Even though they are not in my go.mod I think believe this changed 1.12(?) as I used to have his behaviour. At this point I'm just thinking that these dependencies are listed in my go.sum for testing purposes. It is cool that I can test my code and all dependencies but it is impractical for me because when I do this go test all it fails to compile on some deep package github.com/jackc/pgx and starts testing cockroachdb itself (which is huge).

At this point I start googling and find all kinds of commands in issues such as go mod why all go mod why -vendor, go list or event go list -test -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}'| sort -u to figure how how these dependencies end up in my project. None of them seem to give a conclusive answer so I take a step back:

Want I want to know is which code ends up in the binary I ship. So I can also just look at the build and see which packages are included for that, so I run: go build -o /tmp/my-project -v. I was pretty sure that the -v command would show which packages are being build (help says: "print the names of packages as they are compiled." ) but instead I got empty output... Hmm maybe its using some cache? Sure enough, when I add the -a flag I finally get mostly the information i'm looking for.

Solution If you're looking for the list of packages that:

• Are not (indirect) test dependencies
• Take into account build flags

Simply build your binary using go build -v -a and see which packages are listed there. It also lists standard library packages so it still convoluted with information that you might not be interested in but at least it gives answer to the question.

Ideas It is not that i'm new to programming with Go, and have spent considerate amount teaching the language to (new) engineer. I'm a big fan of the design rational (having read all the blog posts, they are awesome!) but experiences like the one above leave a lot to be desired. I have the following ideas that could solve the (problematic) user story above.

• have a comment in the go.sum file that indicate whether a dependency is indirect and/or just for testing. A I understand is the .sum file acts as the ultimate list of all external dependencies that are needed in one-way or the other and new comers always like to look a files first. Having the information there makes it more accessible
• the go why should always have an answer for each module in .sum file. And specifically without any extra flags. A user user goes like this: "Hey there is a dependency here I didn't expect, why?" and expects to get some sort of answer with "go why "
• Have a command that shows which external packages will be used for a build without the binary actually being build while taking into account build flags. I my case I just wanted to know what external code was shipping with my application.
• Maybe update the FAQ in the wiki with some of the information above?
• Maybe have the -v flag of the build command show when packages are not being build because of the cache?

Hopefully this information is usefully and I know that there are probably complex underlying issues that I don't understand as i'm not very knowledgable about how it all works. But go modules is also a part of the tooling new comers will interact with from the get go (phun intentded), so it makes sense to put some effort in the usability aspects.

Cheers,

Ad

[bcmills]

Want I want to know is which code ends up in the binary I ship.

That's what the -deps flag to go list is for:

go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./my-project | sort -u

• the go why should always have an answer for each module in .sum file

go.sum is not intended for human consumption. However, go mod graph should include the information you're looking for. (See also #27900.)

[advdv]

Thank you for your response. I think in general my feedback here is that, besides making sure that dependencies are present, the module system is also a means for engineers to understand a codebase. Especially in large projects, developers need to understand what parts are included when they open and start working on it. The module system acts as a BoM (bill of materials) in that sense, it acts as the blueprint of a piece of software. In my experience (having tough this practice) the current tooling is ill suited for this activity. I understand that somewhere between:

• go list [....]
• go mod why [...]
• go mod graph [...]
• the go.mod file
• the go.sum file

there is a way of answering these questions but as it is now this information and spread out and confusing.

[flibustenet]

Is there something that prevent to put something like // indirect-test in go.mod ? It's not clear if it's not because we don't want this design or because of a technical difficulty ?

[bcmills]

@flibustenet, not all dependencies of your module are listed in your module's go.mod file. It only includes requirements that satisfy direct imports from your module, or that are needed by a transitive dependency but not required in any dependency's go.mod file.

So a comment in the go.mod file would be actively misleading: it would convey the impression that all indirect dependencies are listed there, and in the long term the vast majority probably won't be.

[akyoto]

Sometimes users of a lib will check go.mod files to see how heavy a given dependency for their project is. If the repository lists 5 binary dependencies and 95 test dependencies it is definitely less scary to use than a package that lists 100 dependencies (numbers exaggerated, but you get the point).

Separating the binary deps and test deps into 2 separate categories (build and test) inside the go.mod file would avoid this confusion.

require.build (
    dep/1
    dep/2
)

require.test (
    dep/3
    dep/4
)

[eclipseo]

I would like to add that in downstream Linux packaging, we would like to have an easy way to split tests dependencies from non-tests dependencies. It is important for us to be able to disable tests and tests deps to solve cyclic dependencies (since we build iteratively, not everything together, it happens often). (speaking as a Go-SIG member for Fedora).

[nim-nim]

To complete what @eclipseo wrote: if go.mod stated something was only needed for tests, it would be possible to configure the CI/CD toolchain to ignore it for builds that are not expected to run tests¹

If the info is not available in go.mod, and you need to re-parse the project tree to get it, why bother with the module system at all? You end up rewriting your own private module-like component handling logic.

And I write go.mod here, but I actually do not care that much about this file. If the info could be requested via golang/x/mod and not written in the go.mod file itself, that would also work for us, though I suspect many would find it terribly inconvenient.

¹ The general case, since the Go definition of a test is lax enough, to allow dependencies on network calls, private servers, private accounts, root access, etc all things that can not be relied to exist outside the original test writer private env. Trying to run a set of Go tests, written by another org, is an exercise in frustration, just to sift real code failures from differences in environment.

[seebs]

I got to thinking about approximately this issue again recently in a context that isn't quite the same as the test case, but similar: Build tags.

Consider a hypothetical build tag, "fancy". If I build with -tags fancy, at least one additional source file is included. It imports something not imported by anything else in the module or its dependencies.

If I'm building with -tags fancy, I have opinions on which version I want to import. If I'm not building with -tags fancy, I don't have opinions on that and probably don't want it downloaded.

But it gets better: Imagine that the tag isn't fancy but internal, and that the module imported/depended on is not generally available. It's not just that it's a waste of time to put it in the module list; it's that no one can download it at all, so if it's present, things break.

I don't see a way to express or handle this cleanly. I don't have any ideas, but I thought it'd be worth mentioning this in what appears to be the most active issue discussing the topic, because it's similar enough that a fix for the other case might address this one too.

[bcmills]

@seebs, the general solution for that case is to load the dependency graph lazily, not to omit dependencies from it altogether.

We're looking at what it would take to facilitate lazy loading of the module graph for 1.15. Hopefully we'll have something to propose not too long after the 1.14 freeze.

[eclipseo]

Can we kindly get an update on this? This is still critical for downstream packaging. We can't upgrade our tooling to include Go modules if we can't spit the test dependencies from the build dependencies.

[thepudds]

@eclipseo Does https://github.com/golang/go/issues/26955#issuecomment-437476523 work for you?

[eclipseo]

@eclipseo Does #26955 (comment) work for you?

As @nim-nim said:

If the info is not available in go.mod, and you need to re-parse the project tree to get it, why bother with the module system at all? You end up rewriting your own private module-like component handling logic.

So no, it's not a solution that we are currently planning. We would somehow rewrite a tool to map the output of go list to the output of go mod, to determine which modules is tests only.

[gopherbot]

Change https://golang.org/cl/220080 mentions this issue: design/36460: add design for lazy module loading

[nim-nim]

So, to recap the issue,

• we have a CI/CD system
• for security reasons, it’s not allowed to trawl the internet, only internal stashes of vetted data
• runs (including Go builds) are done in network-less ephemeral containers populated from this internal stash
• we need to parse go.mod files (either via custom code relying on golang.org/x/mod, or via parsing the output of a go mod foo tool provided by the Go project) to identify the Go artefacts that need to be present in an ephemeral container for a Go build to succeed
• as long as test dependencies are not clearly indicated as such in go.mod or some go mod foo command provided by the Go project, we are forced to populate containers with test dependencies too
• test dependencies are sufficiently problematic that go get ignores them by default. We just want to replicate this mode when populating containers that build Go software

[bcmills]

as long as test dependencies are not clearly indicated as such in go.mod or some go mod foo command provided by the Go project, we are forced to populate containers with test dependencies too

You can discover the package dependency structure using go list with either the -json or -f flags. (Also note that go list populates the module cache on demand as it walks the import graph, so you can also use go list itself to download dependencies — go mod download is a simplified command-line API for simple use-cases.)

So, no: you are not “forced” to populate containers with test dependencies; nor do you “need to parse go.mod files” in order for the build to succeed.

[nim-nim]

We fill an issue to make module usable in offline context.

You reply "you can just use pre-module commands that download things"

This is not constructive (to be polite)

[bcmills]

@nim-nim, the design of modules integrates module operations into existing commands. It is not productive to ignore the possibility of using those commands, since they are part of the supported API for working with modules.

[nim-nim]

@bcmills then suggest things that do not download things and do not require re-parsing the module code.

The source code has already been parsed. That’s what produced go.mod. If we need to reparse it, because you don’t want to produce an inclusive module desing that addresses the needs or others rm -f go.mod and we’ll stop our work to convert to modules.

If your suggestion is to run commands that will download things for us when our use-case is offline then than is also completely useless.

We know about go list. What do you think we’ve been using all those years? If your suggestion is to continue using go list, except it will now do things that are incompatible with our workflow and download things if we make the mistake to enable modules, then we’ll force disable module everywhere forever (we’re already been doing it, except we didn’t want that to get permanent) . And ask all the projects we work with to drop modules and use go dep or any other system that works for our needs.

[mvdan]

@nim-nim I support the general idea of this issue, and have been lurking the thread since it was created. Your latest comments haven't been constructive, and I think that's quickly derailing this thread and lowering the chances that it will get any further attention.

To give everyone the bigger picture here, there are lots of important issues with modules that need to be worked on. And there are just a few people working on them full-time, like @bcmills. Just look at the sheer amount of open issues. They aren't perfect people, but they are doing their best to resolve issues in a way that helps Go in the long run. So, please, be respectful.

[bcmills]

@nim-nim, my suggestion is to either:

1. populate a Go module cache by whatever means you like (go mod download is one way, but there are others), and then set GOPROXY to point to that cache during offline use, or

2. use go list -json and/or go list -json -m to enumerate dependencies, and then set up a secondary module that uses replace directives to redirect those dependencies in the local filesystem.

Once you have tried either or both of those options, show us the concrete, actionable problems you encountered (problems, not prescriptive solutions!), and we can figure out how to address them in a way that is consistent with the rest of the design of the go command and Go modules.

[nim-nim]

@bcmills as we told you repeatedly for more than a year we already perform code analysis with golist and we already set up a cache using GOPATH and the result of the code analysis. And we’ve done tens of thousands of production builds of hundreds of Go projects over the years using this code. It works and we understand why it works and what are its limitations.

However, we do not like maintaining tooling with behaviour that may differ or not from the behaviour of tooling other people had to write for the same reasons.

When Go modules were announced it looked like the Go project was serious about obviating the need of such custom tooling and provide upstream primitives that worked for everyone.

I personally invested months of work to identify the minimal requirements to switch our build system modules. And weeks more to report them.

You persist in dismissing this work. Your answer is basically “just continue to use custom go list logic, I’m not interested in your use case, report if it does not work” (except when we do report, you just ask us to go back to step 1).

FINE

GO111MODULES=OFF as far as I’m concerned.

Not going to invest more months of migration to modules just to get back to the custom go list logic stage.

[mvdan]

@nim-nim once again, be cordial. It is fine to disagree, but being condescending or dramatic (e.g. dismissing modules entirely) is unnecessary. See https://golang.org/conduct.

If you really intend to not participate here anymore, that's your choice.

[bcmills]

@nim-nim, the numerous issues that you filed are prescribing specific API changes. Specific API changes should go through the proposal process, and in general must meet a high bar — especially if they address use-cases that can already be handled using the existing API.

I am suggesting that you will likely have more success if you report use-cases instead of specific APIs, and make use of the existing APIs already designed for those use-cases when possible. (#35922 is a good example following that model for the use-case of distro packaging specifically. Note that that issue is an ongoing conversation, and we have not yet decided what — if any — additional API may be needed to address that use-case.)

[bcmills]

At any rate, this discussion has clearly wandered far from the topic of viewing non-test dependencies. I'm going to hide some comments to get it back on track.

[thepudds]

Hello @nim-nim and @eclipseo, one quick comment is that if you haven't already read this, it is probably worth reading https://github.com/golang/go/issues/29410#issuecomment-515203754 and some of the related discussion in #29410 around Go modules packaging approaches for Debian. Debian is distinct from Fedora, so that might not be useful, but maybe there is some overlap at least in terms of some of the challenges. There is also a short comment there from @qbit (who I think works on packaging for OpenBSD) at https://github.com/golang/go/issues/29410#issuecomment-527678401. There is also an older discussion on OpenBSD packaging challenges with modules in #26852. Again, perhaps not useful, but wanted to at least mention it.

I have followed some of the different issues you have submitted, but not all of them, so sorry if this is redundant or otherwise not helpful.

[davidkhala]

I am laymen. I simply look for when ater I run go mod tidy, those package apparently used only within *_test.go file will not be external

[komuw]

hubris got the better of me and I created: https://github.com/komuw/ote which adds a // test comment next to test dependencies

[tim-gp]

It would be useful if there was a way to get an equivalent list with go list as you get from go version -m binary. If you are planning to distribute the binary, it's valuable to know this list quickly perhaps without having to run a build.