cmd/go: `go mod download -json` is incomplete

Stebalien commented 5 years ago

What version of Go are you using (`go version`)?

$ go version
go version go1.11.4 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/steb/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/steb/projects/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/run/user/1000/tmp/go-build023848120=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran go mod download -json.
Bundled up all files listed in the JSON output (adding /list files as needed).
Tried to replay this bundle, using an HTTP server as a GOPROXY.

What did you expect to see?

go build, go mod download should work without fetching anything.

What did you see instead?

go build and go mod download is getting 404 errors due to files missing from the bundle. It looks like go build is trying to fetch:

Packages mentioned in go.sum files that aren't actually needed for the build (as far as I can tell, at least).
Older (minor) versions of packages mentioned in some dependencies' go.mod files.

I'd expect go mod download -json to return a list of every downloaded package.

Context: I'm trying to integrate go mod with IPFS.

Dead-simple (but easy to read) bash implementation: https://gist.github.com/Stebalien/7bfd1ea07f81c1fbbe9eadac4faeccd3
Slightly better (but more complicated) go implementation: https://github.com/Stebalien/ipgo/blob/master/main.go

Testing against: https://github.com/anacrolix/torrent/tree/ipgo

Stebalien commented 5 years ago

It looks like go mod download -json returns all the modules with code needed to build the current package. However, go needs some metadata from all modules mentioned in go mod graph.

Options as I see them:

Allow go build and friends to work when this metadata is unavailable. Given that go doesn't appear to need the code, this should be possible as far as I can tell.
Include required metadata in the output of go mod download -json. That is, return Module structs with the Zip field omitted.
Update the documentation for go mod download -json.

bcmills commented 5 years ago

go build isn't trying to fetch anything extra based on the contents of go.sum, but it is trying to fetch go.mod files for all of the module versions that appear in the requirements of any module in the module graph.

Probably we should download those go.mod files in go mod download and include them in the -json output.

golang / go