cmd/go: add a facility to peek into the module cache

[matloob]

cc: @rsc @bcmills @suzmue @ianthehat

To work with modules, a tool like goimports should be aware of the set of modules in the module cache. Then, it could suggest packages that haven't been required in go.mod or imported by a package or its deps. At the least, it should be able to list the latest version of each module in the user's module cache. (And if GOPROXY is set, it should be able to list the latest version of the modules served by the proxy.)

Currently there doesn't seem to be a way for a tool like goimports to suggest packages outside the transitive module dependencies of the current module without guessing the module cache directory (which is changing: golang.org/cl/126755) and then crawling through the directories. It seems like the directory layout may be stable but finding the directory is a bit dodgy. And we'd need to handle goproxy manually too.

go/packages has been using go list to find metadata about packages, so ideally we could use go list or maybe go mod to get at this information. But maybe there's a better way to get at it.

[myitcv]

Just a quick question, because I'm unclear on what I think the answer "should" be here.

How would you expect goimports to respond in the presence of multiple (major) versions of a module (and therefore the packages it contains)?

It would seem to me the search space for goimports as it currently works increases massively as a result of modules.

Would a compromise starting point be that goimports works with the modules referenced from go.mod?

[ianthehat]

The trouble is, that's a strictly worse experience than the existing one...

Currently, if you start a new project, you make a new directory and type some code. When you run goimports, it gets to see all the packages you have ever used, and pick your imports based on those, which is mostly likely to contain the ones you want because most people work on similar things using similar packages. You are also likely to have used go get to check out some examples of the kinds of things you want to do, which will also prime the search space.

In the world of modules, all that information disappears, each new module starts with no imports, and so goimports will not do anything for you.

The search space is not that large, you don't need to worry at all about minor versions, you just need to know what import paths have entries in the module cache, you will check whatever version would be added by mvs if the import is added to the file, because that is what you will be doing, goimports will never write the module file for you.

Major versions are the same problem go imports would currently face if you checked out the modules to your go path anyway, two different packages of the same name, pick the best one, something it already has to do. We probably have to add some logic to bias towards larger major versions when you have multiple fully matching candidates, but we would have to do that even if you were still using GOPATH rather than the module cache.

[matloob]

Yes, to echo @ianthehat, goimports using only the modules refrenced from go.mod would be a major regression for users.

As I understand it, different major versions of a module are considered as distinct modules in pretty much every way. And goimports might want to add some heuristic that might prefer the later major version when both are in the module cache, but it's mostly the same problem as a fork of a module, or (in the GOPATH world) the fork of a package.

[sam3d]

(related #24661)

[bcmills]

Would #26718 address all of the known use-cases for this, or are there others?

[bcmills]

CC @heschik

[bcmills]

32337 suggests GOPROXY=off as one way we could allow users to explicitly request to resolve packages and/or module versions using only the existing contents of the module cache.

[dolmen]

43646 suggests to add GOPROXY=cache for a distinct behaviour from GOPROXY=off.

[dolmen]

Note that the directory structure of the module cache is officially documented at https://go.dev/ref/mod#module-cache. Can we interpret this as some stability guarantee?