notebook doesn't work with tornado 5, but doesn't prevent installation

[fterbo]

The notebook installation via pip just requires tornado>4, which will install tornado 5 and thus will not launch:

AttributeError: type object 'IOLoop' has no attribute 'initialized'

In order to make fresh jupyter installs work right now you have to install tornado<5 by hand, and then install jupyter.

[takluyver]

Like this? You need to upgrade pyzmq. Notebook should already be compatible with tornado 5.

[fterbo]

It seems like the same problem, but we install 100% fresh on a new ubuntu system (xenial) just using pip.

[minrk]

A fresh install with pip will get pyzmq 17. Do you have the commands you ran? Here's a Dockerfile that gets a working notebook with pyzmq 17 and tornado 5.0.1:

FROM ubuntu:16.04
RUN apt-get -y update && apt-get install -y python3-pip
RUN pip3 install notebook && pip3 freeze
ENV HOME /tmp
USER 1000
CMD ["jupyter", "notebook", "--ip=0.0.0.0"]

The output of pip freeze running this today:

bleach==2.1.3
decorator==4.2.1
entrypoints==0.2.3
html5lib==1.0.1
ipykernel==4.8.2
ipython==6.2.1
ipython-genutils==0.2.0
jedi==0.11.1
Jinja2==2.10
jsonschema==2.6.0
jupyter-client==5.2.3
jupyter-core==4.4.0
MarkupSafe==1.0
mistune==0.8.3
nbconvert==5.3.1
nbformat==4.4.0
notebook==5.4.1
pandocfilters==1.4.2
parso==0.1.1
pexpect==4.4.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
ptyprocess==0.5.2
Pygments==2.2.0
python-dateutil==2.7.0
pyzmq==17.0.0
Send2Trash==1.5.0
simplegeneric==0.8.1
six==1.11.0
terminado==0.8.1
testpath==0.3.1
tornado==5.0.1
traitlets==4.3.2
wcwidth==0.1.7
webencodings==0.5.1

Note that pip doesn't upgrade dependencies you already have, so if you already had pyzmq 16, it would not be upgraded, though it would need to be.

Unfortunately, there is no way to express that the notebook needs pyzmq 17 if you have tornado 5. Package managers don't seem to have this kind of conditional dependency, so we can only solve it at the documentation level: if you upgrade tornado to 5.0, you will need to ensure several other packages are up-to-date, specifically jupyter-client, ipykernel, notebook, qtconsole, pyzmq (at least).

[fterbo]

The vagrantfile that builds our environment has these substantive parts for installing jupyter:

sudo http_proxy=#{cache} apt-get install -y --no-install-recommends wget build-essential python2.7 python-dev graphviz libssl1.0.0 python-kerberos git
wget -q https://bootstrap.pypa.io/get-pip.py
sudo python2.7 get-pip.py
sudo pip2 install --upgrade six
sudo pip2 install graphviz functools32 "ipython<6.0.0"
sudo pip2 install jupyter

I will need to build one from scratch and freeze it (although since the dependency versions constantly change, it's entirely possible the problem goes away at some point too, if we are getting zmq from something else).

I suppose we probably more generally need to be in the business of pinning versions instead of letting them float and creating subtly different environments based on the date on which you did an install.

[minrk]

Is there a chance your http_proxy=#{cache} means that you are using an outdated cached response from pypi that tells you what the latest version of pyzmq is? I could imagine that results in getting less-than-up-to-date versions of some packages, depending on cache behavior.

Freezing environments is indeed a good idea. We've picked up a practice in repo2docker largely derived from the javascript community of specifying our 'loose' requirements (this is what we should need), but always storing a frozen env and using that for all installation, so that secondary dependencies aren't allowed to float. This allows you to upgrade what's actually installed only through explicit actions of "refreezing" the environment by performing a fresh install from the loose env and saving the result of pip freeze.

[minrk]

This vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/artful64"

  config.vm.network "forwarded_port", guest: 8888, host: 9999

  config.vm.provision "shell", inline: <<-SHELL
    sudo apt-get -y update
    sudo apt-get install -y --no-install-recommends wget build-essential python2.7 python-dev graphviz libssl1.0.0 python-kerberos git
    wget -q https://bootstrap.pypa.io/get-pip.py
    sudo python2.7 get-pip.py
    sudo pip2 install --upgrade six
    sudo pip2 install graphviz functools32 "ipython<6.0.0"
    sudo pip2 install jupyter
  SHELL
end

results in the following pip2 freeze which includes a working notebook with everything up to date:

backports-abc==0.5
backports.shutil-get-terminal-size==1.0.0
bleach==2.1.3
configparser==3.5.0
decorator==4.2.1
entrypoints==0.2.3
enum34==1.1.6
functools32==3.2.3.post2
futures==3.2.0
graphviz==0.8.2
html5lib==1.0.1
ipykernel==4.8.2
ipython==5.5.0
ipython-genutils==0.2.0
ipywidgets==7.1.2
Jinja2==2.10
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
MarkupSafe==1.0
mistune==0.8.3
nbconvert==5.3.1
nbformat==4.4.0
notebook==5.4.1
pandocfilters==1.4.2
pathlib2==2.3.0
pexpect==4.4.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
ptyprocess==0.5.2
Pygments==2.2.0
pykerberos==1.1.5
python-dateutil==2.7.0
pyzmq==17.0.0
qtconsole==4.3.1
scandir==1.7
Send2Trash==1.5.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.11.0
terminado==0.8.1
testpath==0.3.1
tornado==5.0.1
traitlets==4.3.2
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==3.1.4