cmd/go: [modules + integration] go mod requires, list the direct requirements for using a module

[nim-nim]

This report is part of a series, filled at the request of @mdempsky, focused at making Go modules integrator-friendly.

Please do not close or mark it as duplicate before making sure you’ve read and understood the general context. A lot of work went into identifying problems points precisely.

Needed feature

Go needs an official go mod requires command that processes packed Go module file and returns the list of dependencies needed to use this module in other code.

go mod requires could be implemented as a go mod graph mode, via specific option flags, or as a separate subcommand.

It is the pendant of go mod buildrequires (#31300) and used by the CI/CD system to populate the job runtime environment, after go mod buildrequires analysis.

Constrains

• the output should be a machine-readable list of:
  • module path,
  • minimal version
• the feature should also be exposed as a function in the official Go API, returning a go object (map, list, struct…)
• the output should be either be pre-filtered, or be easily filterable, without further go tooling invocations, by:
  • GOOS (only return results for this GOOS),
  • GOOS+GOARCH (only return results for this GOOS and GOARCH), or
  • complete build context (GOOS+specific build flag set)
• the output should be either be pre-filtered, or be easily filterable, without further go tooling invocations, by direct and indirect requirements
• the output should be either be pre-filtered, or be easily filterable, without further go tooling invocations, by:
  • code reuse requirements (modules needed to use the code in other projects)
  • compiler test requirements (modules needed to run tests, that only require the compiler)
  • integration test requirements (modules needed to run tests, that require more than just the compiler to run, and may not be satisfied by the average CI/CD system):
    • internet access,
    • root access
    • some specific software instance
    • some credentials, etc
  • the command should work in secure no-download no-cache mode. In that mode it would probably restricts its output to direct dependencies.

Motivation

• this is pretty much what go mod graph already does, except that this issue asks for a finer-grained dependency view. So it would be a natural go mod graph extension, and only requires a separate subcommand if there is a wish to keep go mod graph output holistic and unfiltered.

• the finer grained view is needed because populating a CI/CD build environment with more code, than the strict minimum the run will need, gets prohibitively expensive in run time, for busy build farms with a huge list of jobs to run. Therefore the build requirement list needs to be as cut down as possible, allowing to remove the requirements unneeded on a particular GOOS/GOARCH. GOOS/GOARCH is the finest filtering mode, that allows module reuse. GOOS/GOARCH is a constrain shared by all the other code that may wish to reuse the module. That is not the case of other project-specific build tags.

• in integration mode, any go module that already went through the CI/CD integration pipeline, and has been accepted in the project baseline, already passed all its unit tests, therefore injecting the dependencies of those unit tests in jobs, that reuse this module, is unnecessary and undesirable.

• missing modules will typically be populated from the organisation baseline. Because this baseline can be shared between organisation projects, it won't be mirrored in each project VCS in a vendor directory.

• needed modules, identified by the go mod requires call, will typically be populated from the organisation baseline.

• adding new modules to the organisation baseline is a lot of work². It requires

  • assigning a curating team,
  • sifting between forks to identify the actual current project upstream,
  • sifting between VCS mirrors to find the root VCS,
  • legal analysis,
  • test result analysis,
  • writing the corresponding recipe for the CI/CD system
  • etc.

• therefore, there was violent disbelief and rejection among consulted integrators³, of any Go module setup, that forced them to process new modules just because the corresponding imports exists in module parts they have no use for:

  • dependencies of GOOS/GOARCH-specific code, when this GOOS/GOARCH is not part of the organisation targets
  • dependencies of upstream integration tests, that have no chance to ever run in our CI/CD setup, because they use elements not available in it (typically, direct internet calls or root access)
  • dependencies of example code, that will never be used in production, and typically does not compile because its upstream is not keeping it up to date
  • their GOPATH/vendor experience of third party code has made them extremely sensible to this kind of spurious import, and the amount of work it represents.

[thepudds]

FYI, as of now, there are currently 7 or so related issues filed today. See for example this search on open issues authored by @nim-nim.

CC @bcmills @rsc

[nim-nim]

@thepudds Thanks for the ping. I still have a few more to write, it takes time to put things into order and write a coherent report.

[nim-nim]

@thepudds @bcmills @rsc Since the generic issue boilerplate references issue #29452, it has an up-to-date list of all the reports I filled.

Short-term the only remaining showstopper for is issue #31304. I know how to workaround or get by without the others, in degraded mode¹.

Mid to long-term local workarounds and degraded mode are a terrible strategy. Once the language is no longer new and shiny (which will eventually happen to Go), people will actively resist getting involved in something that makes their integration work an obstacle course, and people already involved will migrate to bluer skyes. It’s not a single issue problem, it’s a death by a thousand papercuts problem, where each small tooling misbehaviour and missing functionnality contributes to human rejection.

¹ Workaround means reimplementing the Go tooling:

• walk the filesystem manually and inspect it with build.ImportDir without go list,
• use modfile.Parse instead of relying on go mod,
• rely on my understanding of the documentation of how things are supposed to work instead of using the official implementation,
• and other easy to get wrong dangerous things

[nim-nim]

(also, issue #31304 is something more in @jcajka’s ballpark than mine Fedora-side, since it touches directly the go compiler behaviour)