cmd/go: heisen-dependency on google.golang.org/appengine

[Kubuxu]

What version of Go are you using (go version)?

$ go version
go version go1.12.6 linux/amd64

I've also partially tested go1.13. The build didn't succeed (because of https://github.com/golang/xerrors/pull/2) but the bug shown up either way.

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/kubuxu/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/kubuxu/go/"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/kubuxu/go2/test/go-ipfs/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build559096814=/tmp/go-build -gno-record-gcc-switches"

What did you do?

> git clone https://github.com/ipfs/go-ipfs
> cd go-ipfs
> go mod tidy
> go build ./cmd/ipfs
> cp go.mod go.mod.after-build
> go mod tidy
> cp go.mod go.mod.after-tidy
> diff go.mod.after-build go.mod.after-tidy

To test Go 1.13 use misc/go1.13 branch.

What did you expect to see?

go build should not pull in dependencies into go.mod that go mod tidy latter removes.

What did you see instead?

    google.golang.org/appengine v1.4.0 // indirect

google.golang.org/appengine is pulled in by go build and removed by go mod tidy.

go mod why -m google.golang.org/appengine returns

# google.golang.org/appengine
(main module does not need module google.golang.org/appengine)

Unfortunately I was not able to isolate this to one package.

[bcmills]

Neat bug. Still don't know what's going on, but thanks for the clear steps to reproduce!

CC @jayconrod

[bcmills]

This also reproduces with go mod graph, so it's probably triggered by merely loading the module graph. I suspect it's related to #29773, but have to evidence to confirm that as of yet.

go-ipfs$ git checkout misc/go1.13
Already on 'misc/go1.13'
Your branch is up to date with 'origin/misc/go1.13'.

go-ipfs$ git diff

go-ipfs$ go mod graph | grep appengine
github.com/ipfs/go-ipfs google.golang.org/appengine@v1.4.0
google.golang.org/appengine@v1.4.0 github.com/golang/protobuf@v1.2.0
google.golang.org/appengine@v1.4.0 golang.org/x/net@v0.0.0-20180724234803-3673e40ba225
google.golang.org/appengine@v1.4.0 golang.org/x/text@v0.3.0
golang.org/x/tools@v0.0.0-20190226205152-f727befe758c google.golang.org/appengine@v1.4.0
google.golang.org/grpc@v1.19.0 google.golang.org/appengine@v1.1.0

go-ipfs$ git diff
diff --git i/go.mod w/go.mod
index 3c3ac7896..f4ae8fa06 100644
--- i/go.mod
+++ w/go.mod
@@ -109,6 +109,7 @@ require (
        go4.org v0.0.0-20190313082347-94abd6928b1d // indirect
        golang.org/x/sync v0.0.0-20190423024810-112230192c58 // indirect
        golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
+       google.golang.org/appengine v1.4.0 // indirect
        gopkg.in/cheggaaa/pb.v1 v1.0.28
        gotest.tools/gotestsum v0.3.4
 )

[bcmills]

Here's a more minimal reproducer:

go-ipfs$ cat >./imports.go <<EOF
package imports

import (
        _ "github.com/ipfs/go-unixfs"
        _ "github.com/ipfs/iptb-plugins"
)
EOF

go-ipfs$ cat >./go.mod <<EOF
module github.com/ipfs/go-ipfs

require (
        github.com/ipfs/go-unixfs v0.2.0
        github.com/ipfs/iptb-plugins v0.1.0
)

go 1.12
EOF

go-ipfs$ go mod tidy && grep appengine go.mod

go-ipfs$ go list all >/dev/null && grep appengine go.mod
        google.golang.org/appengine v1.4.0 // indirect

Also works if I rename the main module to example.com.

[Kubuxu]

I have a different project (which, unfortunately, I cannot share) where the bug also manifests. It uses go-unixfs but doesn't use iptb-plugins. It leads me to believe that the core of the issues is deeper in the dependency tree.

[bcmills]

Yes, the core of the problem is definitely deep in the graph. Still getting more minimal. Smallest reproducer so far:

module example.com

require (
    github.com/smartystreets/assertions v1.0.0 // indirect
    github.com/smartystreets/goconvey v0.0.0-20190710185942-9d28bd7c0945
    go.opencensus.io v0.22.0
)

go 1.12

package imports

import (
    _ "github.com/smartystreets/goconvey"

    _ "go.opencensus.io"
)

[bcmills]

This is probably a duplicate of #31248.

[bcmills]

It's not an exact repro, but I think I have a case that essentially boils down to the same underlying problem.

env GO111MODULE=on

cd main

go mod graph
stdout i@v0.1.0
! grep 'i v' go.mod

go list all
stdout '^i$'
! grep 'i v' go.mod

go list -m all
stdout '^i v0.1.0$'

-- main/go.mod --
module main

go 1.13

require (
    a v0.0.0
    b v0.0.0
    c v0.0.0
)

replace (
    a => ../a
    b => ../b
    c => ../c
    x v0.1.0 => ../x1
    x v0.2.0 => ../x2
    i => ../i
)
-- main/main.go --
package main

import (
    _ "a"
    _ "b"
    _ "c"
)

func main() {}
-- a/go.mod --
module a
go 1.13
require x v0.1.0
-- a/a.go --
package a
-- b/go.mod --
module b
go 1.13
require x v0.2.0
-- b/b.go --
package b
-- c/go.mod --
module c
go 1.13
-- c/c.go --
package c
import _ "i"
-- x1/go.mod --
module x
go1.13
require i v0.1.0
-- x2/go.mod --
module x
go1.13
-- i/go.mod --
-- i/i.go --
package i

This lists package i in the output of go list all, but fails to list its containing module in go list -m all.

[gopherbot]

Change https://golang.org/cl/186537 mentions this issue: cmd/go/internal/mvs: retain modules required by older versions

[gopherbot]

Change https://golang.org/cl/186557 mentions this issue: cmd/go/internal/mvs: in Req, omit versions implied by older-than-selected versions already in the graph

[bcmills]

Duplicate of #31248