go get gophernotes fails with GO111MODULE=on

[kortschak]

I discovered this trying to set up a my-binder using gophernotes with a repo that has a go.mod file. Apparently my-binder sets GO111MODULE=on when this file exists and the Go version permits.

This is the failure (replicated locally not in $GOPATH):

~ $ echo $GO111MODULE
on
~ $ go get -v github.com/gopherdata/gophernotes
go: finding github.com/gopherdata/gophernotes v0.2.1
go: downloading github.com/gopherdata/gophernotes v0.2.1
go: extracting github.com/gopherdata/gophernotes v0.2.1
go: finding github.com/alecthomas/gozmq latest
go: finding github.com/nu7hatch/gouuid latest
go: finding github.com/gopherds/gophernotes/internal/repl latest
go: downloading github.com/nu7hatch/gouuid v0.0.0-20131221200532-179d4d0c4d8d
go: extracting github.com/nu7hatch/gouuid v0.0.0-20131221200532-179d4d0c4d8d
go: downloading github.com/alecthomas/gozmq v0.0.0-20140622232202-d1b01a2df6b2
go: extracting github.com/alecthomas/gozmq v0.0.0-20140622232202-d1b01a2df6b2
go: finding github.com/gopherds/gophernotes/internal latest
go: finding github.com/gopherds/gophernotes v0.2.1
go: downloading github.com/gopherds/gophernotes v0.2.1
go: extracting github.com/gopherds/gophernotes v0.2.1
Fetching https://golang.org/x/tools/imports?go-get=1
Fetching https://golang.org/x/tools/go/ast/astutil?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/ast/astutil?go-get=1 (status code 200)
get "golang.org/x/tools/go/ast/astutil": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/ast/astutil?go-get=1
get "golang.org/x/tools/go/ast/astutil": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools/imports?go-get=1 (status code 200)
get "golang.org/x/tools/imports": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/imports?go-get=1
get "golang.org/x/tools/imports": verifying non-authoritative meta tag
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
go: finding github.com/motemen/go-quickfix latest
go: finding golang.org/x/tools/go/ast/astutil latest
go: finding golang.org/x/tools/imports latest
go: downloading github.com/motemen/go-quickfix v0.0.0-20160413151302-5c522febc679
go: extracting github.com/motemen/go-quickfix v0.0.0-20160413151302-5c522febc679
Fetching https://golang.org/x/tools/go/ast?go-get=1
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/ast?go-get=1 (status code 200)
get "golang.org/x/tools/go/ast": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/ast?go-get=1
get "golang.org/x/tools/go/ast": verifying non-authoritative meta tag
go: finding golang.org/x/tools/go/ast latest
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
get "golang.org/x/tools": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools?go-get=1
go: finding golang.org/x/tools latest
Fetching https://golang.org/x/tools/go?go-get=1
go: downloading golang.org/x/tools v0.0.0-20190418235243-4796d4bd3df0
Parsing meta tags from https://golang.org/x/tools/go?go-get=1 (status code 200)
get "golang.org/x/tools/go": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go?go-get=1
get "golang.org/x/tools/go": verifying non-authoritative meta tag
go: finding golang.org/x/tools/go latest
go: extracting golang.org/x/tools v0.0.0-20190418235243-4796d4bd3df0
golang.org/x/tools/internal/semver
github.com/alecthomas/gozmq
golang.org/x/tools/internal/fastwalk
golang.org/x/tools/go/ast/astutil
github.com/nu7hatch/gouuid
github.com/pkg/errors
golang.org/x/tools/go/internal/gcimporter
golang.org/x/tools/internal/module
golang.org/x/tools/go/internal/packagesdriver
golang.org/x/tools/internal/gopathwalk
github.com/motemen/go-quickfix
# github.com/alecthomas/gozmq
pkg/mod/github.com/alecthomas/gozmq@v0.0.0-20140622232202-d1b01a2df6b2/zmq_2_x.go:35:41: could not determine kind of name for C.ZMQ_HWM
pkg/mod/github.com/alecthomas/gozmq@v0.0.0-20140622232202-d1b01a2df6b2/zmq_2_x.go:34:40: could not determine kind of name for C.ZMQ_MCAST_LOOP
pkg/mod/github.com/alecthomas/gozmq@v0.0.0-20140622232202-d1b01a2df6b2/zmq_2_x.go:32:40: could not determine kind of name for C.ZMQ_RECOVERY_IVL_MSEC
pkg/mod/github.com/alecthomas/gozmq@v0.0.0-20140622232202-d1b01a2df6b2/zmq_2_x.go:33:40: could not determine kind of name for C.ZMQ_SWAP
golang.org/x/tools/go/gcexportdata
golang.org/x/tools/go/packages
golang.org/x/tools/imports
github.com/gopherds/gophernotes/internal/repl

It looks like you need to tag a more recent version to ensure that users using modules will get a sufficiently recent version by default.

[cosmos72]

Hi, gophernotes currently uses the vendor directory, which does not mix well with modules. See for example https://www.reddit.com/r/golang/comments/9ai79z/correct_usage_of_go_modules_vendor_still_connects/

[kortschak]

This gives surprising results when packages have a go.mod/go.sum with services like my-binder that set GO111MODULE based on their existence.

Is there an intention to move from vendoring to a modules-based build?

[cosmos72]

I added go.mod and go.sum to upstream gomacro.

To @SpencerPark and @sbinet: what are your opinions about removing the vendor/ directory and declaring instead the exact versions of dependencies in go.mod and go.sum ?

[SpencerPark]

@cosmos72 Learned something new about go11 today and I am really liking it. Is this going to break things on older versions of go? Do we need to keep support for these versions? Can we keep both for a while so there are no issues with go11 using modules but 9 and 10 still have the vendor option?

[cosmos72]

Good point.

Go 1.9 and 1.10 would ignore the files go.mod and go.sum, and will either pick the vendor/ directory (if we keep it) or fetch the latest version of the dependencies (if we drop the vendor/ directory) without any control on which versions they actually fetch.

The latter is definitely not an ideal situation, and it's exactly what vendoring and modules attempt to solve.

Also, I have no idea what happens if you use Go 1.11 with modules together with a vendor/ directory.

[cosmos72]

Update: https://research.swtch.com/vgo-module#end_of_vendoring states that when Go modules are enabled, the vendor/ directory is ignored. That's exactly what's needed for a smooth incremental migration:

• first, add go.mod and go.sum, and also keep the vendor/ directory: users compiling with old Go compilers (or modules disabled) will pick the dependencies from vendor/, while users compiling with new Go compilers and modules enabled will automatically download the exact dependencies versions stated in go.mod and go.sum
• second, when updating dependencies, update go.mod and go.sum in sync with vendor/. This can happen many times during gophernotes development and maintenance
• third, after enough users migrated to module-aware Go compilers, we can decide to drop support for module-unaware Go compilers, and remove the vendor/ directory