Cannot import third party packages

andys0975 commented 5 years ago

I have done 'go get -u gonum.org/v1/gonum/...' and go test without failure. But when I try to do ' import "gonum.org/v1/gonum/mat" ' in gophernotes, I got: repl.go:1:8: error loading package "gonum.org/v1/gonum/mat" metadata, maybe you need to download (go get), compile (go build) and install (go install) it? can't find import: "gonum.org/v1/gonum/mat" and this didn't happen to standard library, only third party library failed

cosmos72 commented 5 years ago

Can you try to execute go install -v -i gonum.org/v1/gonum/mat then launch gophernotes and run again import "gonum.org/v1/gonum/mat" ?

In case it still fails, please send the output of go env to help us pinpointing the problem

andys0975 commented 5 years ago

I tried as you said and still failed. Here is my go env output GOARCH="amd64" GOBIN="" GOCACHE="/home/user/.cache/go-build" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="/home/user/gopath" GOPROXY="" GORACE="" GOROOT="/home/user/go" GOTMPDIR="" GOTOOLDIR="/home/user/go/pkg/tool/linux_amd64" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build711688974=/tmp/go-build -gno-record-gcc-switches"

cosmos72 commented 5 years ago

Hi andys0975,

I forgot to ask the output of go version and a quick description on how you obtained the Go toolchain (from your Linux distribution, from https://golang.org/dl/, compiled from sources, etc). Can you send them too?

The only thing in your go env output that strikes me as odd is GOROOT="/home/user/go": while probably legal in some cases, it can easily cause problems because it means that $HOME/go contains go toolchain instead of the more usual setup, where instead $HOME/go is by default GOPATH, i.e. contains the Go applications/library code you write or download.

I will set up the same Go installation layout and try to reproduce your problem.

andys0975 commented 5 years ago

go version go1.11.1 linux/amd64 (from official binary) My OS is Fedora 28, and the kernel is 4.18.12-200.fc28.x86_64 Thank you BTW, I have changed GOROOT to /usr/local/go before and still failed as well

cosmos72 commented 5 years ago

I am more and more perplexed, because loading a package metadata with go/importer.Default() is such a fundamental operation - Go compiler does it all the time - that the error you report should break Go toolchain too.

Maybe you set some environment variables (GOPATH, GOROOT) differently in your shell and in jupyter ?

akhilravuri1 commented 5 years ago

@andys0975 have you solved this problem. I have this problem too, It picks the standard library packages but it doesn't pick the third-party drivers.

yanchick commented 5 years ago

I have some problem. Anybody found solution for this issue?

cosmos72 commented 5 years ago

Not yet, sorry. I have not been able to reproduce it yet. If you are willing to help by providing details and follow some (hopefully simple) instructions, we can try to debug it cooperatively. No remote access is needed, don't worry.

j0hnsmith commented 4 years ago

@cosmos72 I have this problem too with go 1.13.1 on OS X (installed via .pkg from go downloads page). Happy to help debug it, what do you need?

cosmos72 commented 4 years ago

Hi @j0hnsmith, thanks for the offer!

You mean that after go get -v gonum.org/v1/gonum/mat, if you try in gophernotes

import "gonum.org/v1/gonum/mat"

it still gives an error, right?

Then if you run the shell commands

cd "`go env GOPATH`"
find . -name '*gonum*'

and post their output (please check it does not contain sensitive information), together with the error you receive importing gonum.org/v1/gonum/mat, they could help me to debug the problem.

j0hnsmith commented 4 years ago

I first installed gophernotes about a year ago so I thought I'd clean up everything gophernotes related before I started and reinstall, now the error has gone away :) I think running this after the new go get ... was the key.

cp $GOPATH/src/github.com/gopherdata/gophernotes/kernel/* ~/Library/Jupyter/kernels/gophernotes

encryptblockr commented 2 years ago

why was this closed?

how do we download third party go packages to use in gophernotes? where is documentation explaining this?!?

martinsallandm commented 2 years ago

Hello. I'm sorry to reopen this issue, but I'm still having the same problem as ported before by others. I'm trying to import an external package that works on the terminal but not on the notebook. Thank you!

This is the snippet with the error...

import "gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp"

error loading package "gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp" metadata: error executing "/usr/local/go/bin/go get gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp" in directory "/root/go/src/gomacro.imports/gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp": exit status 1

cosmos72 commented 2 years ago

Well, importing third-party packages at runtime is complicated and requires executing the go toolchain multiple times to download, inspect and compile the package. It makes sense that it's also easy to break.

@martinsallandm in your case, the error is

[...] error executing "/usr/local/go/bin/go get gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp" in directory "/root/go/src/gomacro.imports/gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp": exit status 1

It means that gophernotes tried to execute /usr/local/go/bin/go, which either does not exist or returned an error. Some considerations:

the output of go env executed in the same environment would help
does import "gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp" work in a compiled Go program? I tried go get gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp but it printed an error "404 Not Found"
the same Go toolchain installation that was used to compile gophernotes is needed at runtime to import third-party packages
it's strange that gophernotes tried to execute /usr/local/go/bin/go - maybe it was compiled with a Go toolchain installed in /usr/local/go, which was then uninstalled?

martinsallandm commented 2 years ago

Hello @cosmos72 that you very very much for the answer! it clarifies some important points to me. I'll check the availability of the repository. The package does work outside the notebook. Maybe I have it locally, I'll check. My lack of experience (say absence %-) with go might play a hole on this issue. But thank you very much!! I'll try some more steps and report here what I find. By the way, congratulations on the project! Its very very useful and well done.

cosmos72 commented 2 years ago

When you write "The package does work outside the notebook", what do you mean exactly? (full, compilable code snippet would be appreciated - including the go.mod file).

If you mean that you can write some Go source that does import "gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp" and Go toolchain will successfully compile it, there are a couple of possibilities:

gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp is a local module on your machine, and you have a go.mod file with a "replace" directive that points to its location - this is currently not supported by gophernotes.
gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp is a private repository, and you already have a cached copy of it in $GOPATH/go/pkg/mod - this may be enough for Go toolchain, but gophernotes always executes go get of each third-party package you try to import, so again it is not supported by gophernotes.

encryptblockr commented 1 year ago

what is the use of this project if we cannot download go packages?!?

/home/jovyan/work # go get github.com/adshao/go-binance/v2
# github.com/gorilla/websocket
/go/src/github.com/gorilla/websocket/client.go:411:15: undefined: io.NopCloser
/go/src/github.com/gorilla/websocket/client.go:429:14: undefined: io.NopCloser
/go/src/github.com/gorilla/websocket/conn.go:813:25: undefined: io.Discard
/go/src/github.com/gorilla/websocket/conn.go:1124:11: undefined: io.ReadAll
/go/src/github.com/gorilla/websocket/tls_handshake.go:9:19: tlsConn.HandshakeContext undefined (type *tls.Conn has no field or method HandshakeContext)

cosmos72 commented 1 year ago

Hello @encryptblockr, true, this issue is seriously affecting both gophernotes and gomacro.

There have been various reports, which differ in their details and sometimes also in their cause, but the bulk is "importing third-party packages does not work anymore".

Unluckily, it seems that it's triggered by some change either in Go toolchain or in golang.org/x/tools/go/packages, i.e. the library used to inspect third-party packages.

Since it's probably something outside gophernotes, fixing it may take some time.

(Ah, some of the issues were caused by missing or mismatched Go toolchains - that's a different story).

As a workaround, @larsks found in https://github.com/cosmos72/gomacro/issues/146 that having at runtime a slightly different Go toolchain from the one used to compile gomacro (but it should be the same with gophernotes) fixes this issue.

That's clearly a stop-gap, but it helps - at least until a permanent solution is found. For reference, this is what @larsks wrote in https://github.com/cosmos72/gomacro/issues/146:

The problem appears to be independent of the version of Go used to build gomacro; what matters is the version of Go in $PATH at runtime.

I built and installed gomacro using Go versions 1.19.4, 1.20, and 1.20.7 (installed from upstream release binaries from https://go.dev/dl/), and then tried a third party import with different versions of Go in $PATH.

Here's the test script: [omitted]

And the results:

build: /home/lars/go/bin/gomacro: go1.19.4 runtime: 1.19.4 status: SUCCESS [i.e. can import third-party packages] build: /home/lars/go/bin/gomacro: go1.19.4 runtime: 1.20 status: SUCCESS build: /home/lars/go/bin/gomacro: go1.19.4 runtime: 1.20.7 status: SUCCESS build: /home/lars/go/bin/gomacro: go1.20 runtime: 1.19.4 status: SUCCESS build: /home/lars/go/bin/gomacro: go1.20 runtime: 1.20 status: FAILED [i.e. importing third-party packages fails] build: /home/lars/go/bin/gomacro: go1.20 runtime: 1.20.7 status: SUCCESS build: /home/lars/go/bin/gomacro: go1.20.7 runtime: 1.19.4 status: SUCCESS build: /home/lars/go/bin/gomacro: go1.20.7 runtime: 1.20 status: SUCCESS build: /home/lars/go/bin/gomacro: go1.20.7 runtime: 1.20.7 status: FAILED

Which are...weird. For anything more recent than 1.19.4, the import fails when the runtime version matches the build version.

butuzov commented 5 months ago

works (now) thanks to @cosmos72 !

 go install github.com/gopherdata/gophernotes@5514204
 mkdir -p ~/.local/share/jupyter/kernels/gophernotes
 cd ~/.local/share/jupyter/kernels/gophernotes
 cp "$(go env GOPATH)"/pkg/mod/github.com/gopherdata/gophernotes@v0.7.5/kernel/*  "."
 chmod +w ./kernel.json # in case copied kernel.json has no write permission
 sed "s|gophernotes|$(go env GOPATH)/bin/gophernotes|" < kernel.json.in > kernel.json

cosmos72 commented 5 months ago

Good :)

A posteriori, this seems to have the same underlying cause as gophernotes issue #261 and gomacro issue #146 so fixing them 4 months ago fixed this one too.

The only missing step (on my TO DO list) is releasing a new gophernotes version, that would let users to simply

go install github.com/gopherdata/gophernotes

because at the moment they need

go install github.com/gopherdata/gophernotes@latest

gopherdata / gophernotes

Cannot import third party packages #147