Jupyter image 17.05.1 (quay.io) cannot save notebooks to Galaxy history

[AjitPS]

Hi Bjorn, This is an issue I have been facing lately with Jupyter notebooks in Galaxy. In the past, our Galaxy instance (at Rothamsted) was using the jupyter: 16.01 image in which the Python 2, 3 and R kernels all worked okay and we were able to checkpoint/save notebooks, access Galaxy datasets and also export plots/images to history.

However, we needed to use the DESeq2 R package which wasn't available in the 16.01 image. I tried installing it in the R kernel from Bioconductor via: source("https://bioconductor.org/biocLite.R") biocLite("DESeq2")

But got a "URLs not supported" error. I contacted you in March 2017 about this and you suggested using the latest juypter docker image instead of 16.01.

We are now using that latest image (as of May) but: 1) saving/ checkpoint notebook doesn't work 2) trying to install DESeq2 throws a bunch of errors, similar to https://support.bioconductor.org/p/54381/ : Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘XML’ had non-zero exit status”Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘annotate’ had non-zero exit status”Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘genefilter’ had non-zero exit status”Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘geneplotter’ had non-zero exit status”Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘DESeq2’ had non-zero exit status”Updating HTML index of packages in '.Library'

[AjitPS]

A fix for this seems to be to do: sudo apt-get install libxml2-dev

However, I'm not sure what the best way is for me as a user of the docker image to resolve that.

[AjitPS]

Hi @bgruening , thanks for your suggestion. Switching to using tag: 17.05 now shows a Jupyter token authentication page unlike before. I'm not sure what token it expects the user to provide here. We have a Galaxy-test-Docker VM running Docker containers separate from our Galaxy-test instance VM. It launches the container but I don't get a message with token in the url.

Note: we are using Galaxy 17.01 and this jupyter docker image is tagged 17.05 so I'm not sure if it lacks certain jupyter config files/changes.

[AjitPS]

@bgruening Just to note, I have since spoken to @natefoo and he mentioned that the Galaxy public platform used jupyter:16.01.1 from quay.io and test instance was tested with 17.05 from quay.io. I have since pulled both images from quay.io and noticed that 17.05 gave me the token/authentication page but 16.01.1 didn't. However, installing packages in the R kernel in 16.01.1 again gave the following errors: Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘XML’ had non-zero exit status”Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘annotate’ had non-zero exit status”Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘genefilter’ had non-zero exit status”Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘geneplotter’ had non-zero exit status”Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘DESeq2’ had non-zero exit status”Updating HTML index of packages in '.Library' Making 'packages.html' ... done

To do the above, I launched a new Jupyter notebook, switched to the R kernel and ran: source("https://bioconductor.org/biocLite.R") biocLite("DESeq2")

Trying install.packages("XML") also gives the same error. It seems to be because libxml2-dev is missing but I checked the Dockerfile for Jupyter:16.01.1 and it has a sudo apt-get install for libxml2-dev in there.

[bgruening]

@AjitPS can you please try master or 17.05 again from quay.io. Please don't forget to re-pull.

[AjitPS]

The token authentication is gone! :)

@bgruening But, I get the same package install errors again, when testing this the Galaxy-test instance at Rothamsted:

1. Pulled quay.io/bgruening/docker-jupyter-notebook:master, set it to be used in allowed_images.yml

Note: The public Galaxy is using jupyter-notebook 16.01.1 from quay.io and switching to the R kernel there just causes the kernel to fail and notebook stops.

2. Launched a new Jupyter notebook, switched to R kernel and ran library(DESeq2) , shown as there is no package called ‘DESeq2’.

3. Tried installing via: source("https://bioconductor.org/biocLite.R") biocLite("DESeq2")

4. But get these errors: “installation of package ‘XML’ had non-zero exit status” Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘annotate’ had non-zero exit status” Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘genefilter’ had non-zero exit status” Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘geneplotter’ had non-zero exit status” Warning message in install.packages(pkgs = doing, lib = lib, ...): “installation of package ‘DESeq2’ had non-zero exit status

5. Tried install.packages('XML') as well but get the same error: “installation of package ‘XML’ had non-zero exit status” Warning message in install.packages(pkgs = doing, lib = lib, ...):

[AjitPS]

unable to save Jupyter notebooks as well to Galaxy history, when using jupyter image 16.01.1 or 17.05 or master, from quay.io. @bgruening @natefoo

[AjitPS]

• Did: sudo docker run -i -t quay.io/bgruening/docker-jupyter-notebook /bin/bash

• listed all packages in the image: apt list --installed: which shows: libxml2-dev/now 2.9.1+dfsg1-5+deb8u4 amd64 [installed,local]

[bgruening]

17.05.1 is up now. ping @AjitPS

[AjitPS]

@bgruening I pulled 17.05.1 from quay.io but the issue persists. the image has libxml2-dev 2.9.1 (I checked via docker run) but still can't install DESeq2 (same packages failing due to non-zero exit status). The install fails when trying from within the IE in Galaxy (Jupyter R kernel) or when doing it at command line within the image, e.g, install.packages("XML") fails in both cases.

[AjitPS]

Within the image: > R > install.packages("XML")

[AjitPS]

@bgruening Does it work at your end? Can you install packages within the R kernel via the IE ui or command line??

[bgruening]

Where are you?

[AjitPS]

am upstairs in the training but can come see you at 3pm

[AjitPS]

@bgruening just tried using the new 17.05.1 image and DESeq2 install worked within the notebook in the R kernel. Can now install packages and use them.

install DESeq2:

use lib:

demo:

[AjitPS]

I still have an issue with saving notebooks to history, doesn't seem to work. Also, exporting plot images to history (using python kernel) starts and finishes ok but downloading the plots show blank images.

[AjitPS]

@bgruening @natefoo

Issue summary:

• The Rothamsted Galaxy production instance uses jupyter image 16.01 from docker.io which suggests it exports plots (via put) to user's galaxy history but the plot shows up as a blank image ! It also cannot install packages in Jupyter's R kernel. However, it successfully saves notebooks to history via the save notebook to history button.

• The public Galaxy instance uses jupyter image 16.01.1 which exports plots to history correctly but the save notebook to history button does nothing.

• Rothamsted Galaxy-test now uses jupyter image 17.05.1 from quay.io, as of today, which also exports plots to history correctly. It can now also install packages in Jupyter's R kernel. But, the save notebook to history button is now missing.

[AjitPS]

@bgruening I am now using jupyter image 17.05.1 from quay.io but cannot save notebooks to Galaxy history as the button is missing.

[bgruening]

I will look at this next week OK? On holiday in France :-)

[AjitPS]

Sure, have a nice time in France :)

[AjitPS]

Hi @bgruening , any updates on these Jupyter issues? Thanks

[AjitPS]

@bgruening Also, we updated recently to Galaxy 17.05 but strangely Jupyter IE fails now with this error when trying to launch a notebook: galaxy.visualization.plugins.plugin ERROR 2017-07-13 11:30:02,516 IE plugin request handling failed Traceback (most recent call last): File "/home/usern/galaxy/lib/galaxy/visualization/plugins/plugin.py", line 278, in _render request = self.INTENV_REQUEST_FACTORY( trans, self ) File "/home/usern/galaxy/lib/galaxy/web/base/interactive_environments.py", line 58, in __init__ self.attr.docker_connect_port = int(raw_docker_connect_port) if raw_docker_connect_port else None ValueError: invalid literal for int() with base 10: 'None'

Note: Jupyter worked before on 17.01 but fails after upgrading to Galaxy 17.05

[AjitPS]

@bgruening any suggestions to fix this? in the new Galaxy