Error importing packages from docker build of gophernotes

[cosmos72]

@snowzach reported the following in https://github.com/cosmos72/gomacro/issues/109

I am not expert with docker, so I moved the issue here as it's about gophernotes + docker (I tested on a plain Linux installation of latest gophernotes without docker, and I could not reproduce this issue)

Any suggestion / help is appreciated.

I used go 1.16.3... Here's the Dockerfile I am using to add gophernotes to jupyter. It pulls gophernotes master (which I think is 1.0RC) which depends on github.com/cosmos72/gomacro v0.0.0-20210413180755-590dbc3d0bcd which looks very recent. I also tried forcing module mode but that didn't seem to help.

FROM jupyter/datascience-notebook:latest

# Need to be root to do all this stuff
USER root

# Base image packages
RUN apt-get update \
    && sudo apt-get install -y \
    libpq-dev less \
    && rm -rf /var/lib/apt/lists/*

# Python packages
RUN pip install \
    psycopg2

# Install Go
RUN wget -O /tmp/go.tar.gz https://golang.org/dl/go1.16.3.linux-amd64.tar.gz \
    && tar -C /usr/local -xzf /tmp/go.tar.gz \
    && rm /tmp/go.tar.gz
ENV PATH="${PATH}:/usr/local/go/bin:$HOME/go/bin"
ENV GO111MODULE=on

# Install gophernotes
RUN cd /tmp && git clone https://github.com/gopherdata/gophernotes \
    && cd /tmp/gophernotes && GOPATH=/go go install \
    && mv /go/bin/gophernotes /usr/local/bin/gophernotes \
    && mkdir -p /opt/conda/share/jupyter/kernels/gophernotes \
    && cp kernel/* /opt/conda/share/jupyter/kernels/gophernotes \
    && chmod +w /opt/conda/share/jupyter/kernels/gophernotes/kernel.json \
    && chown -R jovyan:users /opt/conda/share/jupyter/kernels/gophernotes \
    && cd && rm -Rf /tmp/gophernotes && rm -Rf /go

# Switch back to jupyter default user
USER jovyan

When I try using Go it just says:

[I 14:48:07.597 NotebookApp] 302 GET / (172.17.0.1) 0.970000ms
[I 14:48:07.642 NotebookApp] 302 GET /tree? (172.17.0.1) 0.990000ms
[I 14:48:12.580 NotebookApp] 302 POST /login?next=%2Ftree%3F (172.17.0.1) 1.880000ms
[I 14:48:19.120 NotebookApp] Creating new notebook in
[I 14:48:19.208 NotebookApp] Writing notebook-signing key to /home/jovyan/.local/share/jupyter/notebook_secret
[I 14:48:25.080 NotebookApp] Kernel started: 7e12e118-ad92-40a3-87eb-60f398cd57c2, name: gophernotes
[I 14:48:25.209 NotebookApp] Adapting from protocol version 5.0 (kernel 7e12e118-ad92-40a3-87eb-60f398cd57c2) to 5.3 (client).
2021/04/16 14:48:27 Unhandled shell message:  comm_info_request
// debug: running "go get github.com/davecgh/go-spew/spew" ...
go: downloading github.com/davecgh/go-spew v1.1.1
go get: added github.com/davecgh/go-spew v1.1.1

And then the console shows

I did get it to work at one point when I got it to print compiling in that log output.. I never restarted it. It literealy didn't work several times, then I did something and it printed compiling in the log output and after that it seemed to work. Not sure what I did to get that compiling stage to work though.

[snowzach]

This might better be titled Having issues with third party packages and go1.16.3 Docker is just the way I chose to install it.

[cosmos72]

Well, I could not reproduce this issue on the following setups:

• latest gophernotes, compiled on Linux/amd64 with Go 1.16.1
• latest gophernotes, compiled on Linux/amd64 with Go 1.16.3

hence my guess that it's somehow related to the docker installation procedure

[snowzach]

There is really nothing special about the docker installation procedure other than it starts with a clean slate. Would you be able to try with an empty GOPATH? Is it possible you're starting with a cache in place of some sort? Could it matter that this is using the Jupyter Notebook single-image?

[cosmos72]

There is really nothing special about the docker installation procedure other than it starts with a clean slate.

That's what puzzles me.

The sequence to load packages in module mode with Go >= 1.16 is quite convoluted:

1. create a dummy go.mod file in a new, empty directory
2. echo the line // debug: running "go get github.com/davecgh/go-spew/spew" ...
3. run go get pkg/full/path in the directory created at 1
4. call golang.org/x/tools/go/packages/Load("pattern="+pkgpath)
5. create a file pkgshortname.go in the directory created at 1. The file contains all package's symbols retrieved at 4
6. echo the line // debug: running "go mod tidy" ...
7. run go mod tidy in the directory created at 1
8. echo the line // debug: compiling "pkg/full/path" ...
9. run go build -buildmode=plugin in the directory created at 1
10. call plugin/Open() and plugin/Plugin.Lookup() to load and use the plugin

your setup fails at step 4 because the echo at step 6 is not visible in your logs. And golang.org/x/tools/go/packages/Load() is a big black box that, among other things, internally executes the go command.

[butuzov]

@cosmos72 I was able to run third party with no much hassle when using your gomacro 0.7.3 after the recent update. However, I didn't use gophernotes/gophernotes image, but rather mine. And all I change to make it working is to instal build-essentials (gcc/make/etc...).

Docekrfiles can be found here - https://github.com/butuzov/dots/tree/master/docker-images