Not working in module mode

[kortschak]

I am trying to run github.com/kortschak/graphprac on my local machine. I always run with module mode on, Go version 1.13.

When I try to run the first cell of the notebook at that repo I get a failure that the build with buildmode=plugin failed. When I execute this in a terminal to investigate I see that it's because the go tool expects the go.mod file to exist and bails otherwise. Creating a go mod file in that directory allows the build to complete and also allows the cell to execute on the notebook.

Perhaps the go.mod from imported packages' modules could be placed in the gomacro build paths.

At the very least, it should be noted that it might be necessary to invoke the server with GO111MODULE=off jupyter notebook.

[kortschak]

Note that gomacro as of cosmos72/gomacro@24fc92039372d7fcbaf7d715cc5cd6668e5650c0 executes the code that fails here without issue.

[mattn]

OT: gophernotes dropped support of Go 1.10. So we can remove vendor.

[cosmos72]

https://github.com/cosmos72/gomacro has a go.mod and is thus compiled with modules support, but its runtime is not yet module aware - thus evaluating import will not find modules in $GOPATH/pkg/mod.

I am working to fix that.

[cosmos72]

I added modules support to gomacro - i.e. import is now module aware. ASAP I will update also gophernotes to use the latest gomacro.

[cosmos72]

Working on it, see https://github.com/gopherdata/gophernotes/pull/189 Also see #187 for a very similar issue

[cosmos72]

It should work now, waiting for @kortschak confirmation before closing

[mattn]

Question, when importing different version of module which is already used in gophernotes's dependencies, what you expect?

error loading plugin "/home/mattn/go/src/gomacro.imports/gonum.org/v1/plot/plot.so": plugin.Open("/home/mattn/go/src/gomacro.imports/gonum.org/v1/plot/plot.so"): plugin was built with a different version of package golang.org/x/image/math/fixed (previous failure)

[cosmos72]

I expect exactly the error you report, because plugin allows loading the same package multiple times only if they have exactly the same version.

I already thought about this problem in the past, and there are two complementary solutions:

1. If gophernotes depends on a package foo, generate an import file for it in the 'imports' directory
2. When compiling a plugin to load the symbols of a package bar, also cache the symbols of all its dependencies - a later import of one of the dependencies will be satisfied from the cache, without loading another plugin (which may have a different version, as you point out).

These are future improvements for gophernotes (1) and gomacro (2)

[kortschak]

This is fixed for me as noted in the PR.

[mattn]

If I want to use packages in the document and the packages are used in gophernotes, I should set GO111MODULE=off ?

[cosmos72]

Environment GO111MODULE is now ignored by gophernotes at runtime: internally, it always sets it to on before importing a package. The underlying Gomacro has a user-controlled option to toggle GO111MODULE on/off but gophernotes has no mechanism to invoke it (yet).

Anyway there should be very few packages used in gophernotes without a corresponding import file in 'imports' directory: at first glance, I'd say only github.com/pebbe/zmq4 and github.com/satori/go.uuid. Their transitive dependencies are all in the standard library, and that's not an issue.

Maybe I am missing something?

[mattn]

I could solve issue that can't import packages which is used in gophernotes.

One another issue is that can't import packages in different versions. I'm trying to use my package https://github.com/mattn/go-pairplot and this import golang.org/x/image but gonum.org/v1/plot/vg also import golnang.org/x/image/font/fixed.

[mattn]

Hmm, importing "github.com/mattn/go-runewidth" fail.

[sbinet]

Not sure if that's related but 'gopls' also fails to import 'runewidth' (this happens when trying to 'gorename' something in one of my packages that happens to import a pkg that, down the line, imports that 'runewidth' package.)

[mattn]

Hmm, I played using go-runewidth for a little while but I don't see the fail.

[mattn]

Anyone could you please try import "github.com/mattn/go-plotlib" ?

[cosmos72]

import "github.com/mattn/go-plotlib" fails with

error loading plugin "/home/max/go/src/gomacro.imports/github.com/mattn/go-plotlib/go-plotlib.so": plugin.Open("/home/max/go/src/gomacro.imports/github.com/mattn/go-plotlib/go-plotlib"): plugin was built with a different version of package github.com/mattn/go-runewidth

The reason is the following: the latest tagged version of github.com/mattn/go-plotlib is 0.0.6 i.e. commit faf47dabfcfc44f71e7bf404f8effc6ac29a6cc9, which depends on github.com/mattn/go-runewidth 0.0.7.

But gophernotes depends on the most recent tagged version of github.com/peterh/liner i.e. 1.1.0 which depends on github.com/mattn/go-runewidth 0.0.3.

So github.com/mattn/go-runewidth 0.0.3 is statically linked inside gophernotes, thus plugin refuses to load github.com/mattn/go-runewidth 0.0.7.

The only real solution is for @peterh to release a newer version of his github.com/peterh/liner that will depend on github.com/mattn/go-runewidth 0.0.7

[peterh]

I'm disinclined to release a new version of liner every time a 3rd party package sneezes. Fortunately, it appears that this is not "the only real solution" for @cosmos72 .

If I understand the Go module system correctly, "0.0.3" really means "0.x.y, where x >= 0 and y >= 3". So it should allow you to specify require github.com/mattn/go-runewidth 0.0.7 //indirect in your go.mod file (even if you don't actually use runewidth yourself) in order to get the specific version you require. See https://blog.golang.org/using-go-modules#TOC_4.

[cosmos72]

Thanks for the suggestion, I will try it. I hope to be mistaken, because I remember about versions v0.x.y being treated as incompatible - maybe such rule exists in some other system?

[cosmos72]

I updated gomacro/go.mod to explicitly require github.com/mattn/go-runewidth 0.0.7 and it works. I also updated gophernotes/go.mod to require the latest gomacro containing the change above. Thanks @peterh

[cosmos72]

About the other issues listed above:

• import "github.com/kortschak/graphprac" now works
• import "gonum.org/v1/plot" now works too
• import "github.com/mattn/go-pairplot" now works except if you already imported gonum.org/v1/plot in the same session, because they depend on different versions of golang.org/x/image/math/fixed. I don't know whether this is fixable by modifying only gophernotes - suggestions?

[RafalSkolasinski]

I got similar issue when trying

import "github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1"
import "github.com/ghodss/yaml"

I get

error loading plugin "/home/rskolasinski/go/src/gomacro.imports/github.com/ghodss/yaml/yaml.so": plugin.Open("/home/rskolasinski/go/src/gomacro.imports/github.com/ghodss/yaml/yaml"): plugin was built with a different version of package gopkg.in/yaml.v2

It seems it tries to load from $GOPATH/src so not inmodules mode. Is there a way to force usage of $GOPATH/pkg/mod when notebook is in directory containing go.mod file?

Sorry for bumping old issue - can open new one if this is some new problem.

[cosmos72]

Gophernotes should ignore the contents of current directory when importing packages: the import machinery is executed in a newly created directory `$GOPATH/src/gomacro.imports/PACKAGE/FULL/PATH'

Maybe both packages depend on gopkg.in/yaml.v2 - and on different versions of it? The import machinery is not yet smart enough to tell Go compiler "I already loaded version V1 of package X. When getting/compiling package Y which depends on X too, please try to use version V1 of package X instead of the default version, and also instead of the versioned dependency specified by package Y"

[RafalSkolasinski]

Gophernotes should ignore the contents of current directory when importing packages: the import machinery is executed in a newly created directory `$GOPATH/src/gomacro.imports/PACKAGE/FULL/PATH'

Coming to Go from Python background, I used to do a lot of my development by writing Python code in .py files (even packages) and importing it directly in the notebook for testing how it works and fooling around.

In some regards you could say that Jupyter Notebook was my IDE. I hoped that Gophernotes could serve the same purpose for Go. Wonder if worth opening another issue for it?

[cosmos72]

There should be an open issue that, among other things, proposes to implement import "." to load the Go package in the current directory. Unluckily I cannot find it anymore.

Feel free to open a new issue on this topic. If you can explain in detail the behavior you expect, even better :)

[mattn]

Sorry off topic. Can I import specific version of package in notebook?

[cosmos72]

Sorry off topic. Can I import specific version of package in notebook?

Better to discuss this in a separate issue - I opened #218