Minimal working environment specification

[alexhrescale]

Can you please share the minimal working environment that you use for the examples?

Currently, I am running with this environment

$ pip list
Package                           Version
--------------------------------- -------
asn1crypto                        0.24.0
atomicwrites                      1.3.0
attrs                             19.1.0
backcall                          0.1.0
bash-kernel                       0.7.1
bcrypt                            3.1.7
bleach                            3.1.0
cffi                              1.12.3
cryptography                      2.7
decorator                         4.4.0
defusedxml                        0.6.0
entrypoints                       0.3
importlib-metadata                0.19
ipykernel                         5.0.0
ipyparallel                       6.2.4
ipython                           7.0.0
ipython-genutils                  0.2.0
ipywidgets                        7.5.1
jedi                              0.15.1
Jinja2                            2.10.1
jsonschema                        3.0.2
jupyter                           1.0.0
jupyter-client                    5.2.0
jupyter-console                   6.0.0
jupyter-contrib-core              0.3.3
jupyter-contrib-nbextensions      0.5.1
jupyter-core                      4.5.0
jupyter-highlight-selected-word   0.2.0
jupyter-latex-envs                1.4.6
jupyter-nbextensions-configurator 0.4.1
jupytext                          1.2.1
lxml                              4.4.1
MarkupSafe                        1.1.1
metakernel                        0.20.0
mistune                           0.8.4
more-itertools                    7.2.0
nbconvert                         5.4.1
nbformat                          4.4.0
notebook                          5.7.0
packaging                         19.1
pandocfilters                     1.4.2
paramiko                          2.4.2
parso                             0.5.1
pexpect                           4.7.0
pickleshare                       0.7.5
pip                               18.1
pluggy                            0.12.0
plumbum                           1.6.7
prometheus-client                 0.7.1
prompt-toolkit                    2.0.9
ptyprocess                        0.6.0
py                                1.8.0
pyasn1                            0.4.6
pycparser                         2.19
Pygments                          2.4.2
PyNaCl                            1.3.0
pyparsing                         2.4.2
pyrsistent                        0.15.4
pytest                            5.1.0
python-dateutil                   2.8.0
PyYAML                            5.1.2
pyzmq                             18.1.0
qtconsole                         4.5.3
Send2Trash                        1.5.0
setuptools                        41.1.0
simplegeneric                     0.8.1
six                               1.12.0
sshkernel                         0.6.2
terminado                         0.8.2
testpath                          0.4.2
tornado                           6.0.3
traitlets                         4.3.2
wcwidth                           0.1.7
webencodings                      0.5.1
widgetsnbextension                3.5.1
zipp                              0.5.2

The library versions of the dependencies should be matching the lowest working versions as declared in requirements.txt. I installed them using pip install -r reqs.txt where reqs.txt contains

ipykernel==5.0.0
ipython==7.0.0
jupyter_client==5.2.0
metakernel==0.20.0
nbconvert==5.4.1
paramiko==2.4.2
plumbum
PyYAML

where every >= version is replaced with ==. In a blank ipython notebook with the SSH kernel, it fails with e.g.

[W 15:55:21.129 NotebookApp] Timeout waiting for kernel_info reply from 2abdb200-d53d-4fe7-b92f-d744f922ebe0

Another check is by running

python -m sshkernel --debug  # outputs kernel-54116.json

from one terminal in the venv, then running e.g.

jupyter console --debug --existing kernel-54116.json  # taken from above

in another terminal that uses the same venv. This fails with

RuntimeError: Kernel didn't respond to kernel_info_request

Thanks for your help

[m-ueno]

Hi @alexhrescale, thank you for being interested in my work. If you want to try it quickly, please consider using Dockerfile.

Please look the latest passing CI logs, which creates the minimal working environment with pip install command: https://gitlab.com/m-ueno/sshkernel/-/jobs/274535122

...
Successfully installed MarkupSafe-1.1.1 PyYAML-5.1.2 asn1crypto-0.24.0 backcall-0.1.0 bcrypt-3.1.7 bleach-3.1.0 cffi-1.12.3 cryptography-2.7 decorator-4.4.0 defusedxml-0.6.0 entrypoints-0.3 ipykernel-5.1.2 ipyparallel-6.2.4 ipython-7.7.0 ipython-genutils-0.2.0 jedi-0.15.1 jinja2-2.10.1 jsonschema-3.0.2 jupyter-client-5.3.1 jupyter-core-4.5.0 metakernel-0.24.2 mistune-0.8.4 nbconvert-5.6.0 nbformat-4.4.0 pandocfilters-1.4.2 paramiko-2.6.0 parso-0.5.1 pexpect-4.7.0 pickleshare-0.7.5 plumbum-1.6.7 prompt-toolkit-2.0.9 ptyprocess-0.6.0 pycparser-2.19 pygments-2.4.2 pynacl-1.3.0 pyrsistent-0.15.4 python-dateutil-2.8.0 pyzmq-18.1.0 sshkernel-0.9.1 testpath-0.4.2 tornado-6.0.3 traitlets-4.3.2 wcwidth-0.1.7 webencodings-0.5.1

It looks you have old version of sshkernel==0.6.1. How about uninstalling/upgrading sshkernel? Or, how about recreating virtual environments and install it with clean dependency?

[alexhrescale]

Thanks for your reply. I am running in a clean virtualenv (via nix sandbox), and by default, it installs sshkernel==0.9.1, which did not work. Due to #13 I am suspecting there may be regressions in other libraries even in patch version updates.

I have tried the Dockerfile approach, and it works, but I would also like to be able to add sshkernel to an existing jupyter notebook setup. I will use the Docker version as a starting point to find the divergence.

[alexhrescale]

I have a suspicion that the base libraries that are provided from a conda install are the source of the divergence. For the record, these libraries are installed on a fresh Miniconda (conda 4.7.10) install

    - _libgcc_mutex==0.1=main
    - asn1crypto==0.24.0=py37_0
    - bzip2==1.0.8=h7b6447c_0
    - ca-certificates==2019.5.15=0
    - certifi==2019.6.16=py37_0
    - cffi==1.12.3=py37h2e261b9_0
    - chardet==3.0.4=py37_1
    - conda-package-handling==1.3.11=py37_0
    - conda==4.7.10=py37_0
    - cryptography==2.7=py37h1ba5d50_0
    - idna==2.8=py37_0
    - libarchive==3.3.3=h5d8350f_5
    - libedit==3.1.20181209=hc058e9b_0
    - libffi==3.2.1=hd88cf55_4
    - libgcc-ng==9.1.0=hdf63c60_0
    - libstdcxx-ng==9.1.0=hdf63c60_0
    - libxml2==2.9.9=hea5a465_1
    - lz4-c==1.8.1.2=h14c3975_0
    - lzo==2.10=h49e0be7_2
    - ncurses==6.1=he6710b0_1
    - openssl==1.1.1c=h7b6447c_1
    - pip==19.1.1=py37_0
    - pycosat==0.6.3=py37h14c3975_0
    - pycparser==2.19=py37_0
    - pyopenssl==19.0.0=py37_0
    - pysocks==1.7.0=py37_0
    - python-libarchive-c==2.8=py37_11
    - python==3.7.3=h0371630_0
    - readline==7.0=h7b6447c_5
    - requests==2.22.0=py37_0
    - ruamel_yaml==0.15.46=py37h14c3975_0
    - setuptools==41.0.1=py37_0
    - six==1.12.0=py37_0
    - sqlite==3.29.0=h7b6447c_0
    - tk==8.6.8=hbc83047_0
    - tqdm==4.32.1=py_0
    - urllib3==1.24.2=py37_0
    - wheel==0.33.4=py37_0
    - xz==5.2.4=h14c3975_4
    - yaml==0.1.7=had09818_2
    - zlib==1.2.11=h7b6447c_3
    - zstd==1.3.7=h0b5b093_0

Here are steps to get it running, from outside the docker container.

1. from the Docker container, run pip freeze > jovyan.txt

2. from a fresh conda env, run for dep in $(cat jovyan.txt); do pip install $dep --force-reinstall --no-cache; done. This has 2 effects:

   • it installs from pypi, bypassing any conda channel (not all deps are in conda)
   • it resumes even when pip installs fail. Pip installs notably fail when a preinstalled version via distutils cannot be removed (it says distutils but is most likely the conda-provided version).

Note that from within a conda env, the stalling in #13 is not observed. The SSH notebook launches and the kernel launches (but I cannot get the SSH to connect, observed in #15).

Thus, as of now, it would appear that successful use of this library requires conda.

[m-ueno]

Hmm, I think it doesn't require conda.

My develop environment on Ubuntu 18.04 is created with pyenv not conda. Another user successfully running it on Ubuntu 18.10 with OS bundled python https://packages.ubuntu.com/cosmic/python3 . Recent passing continuous integration tests run without conda (see https://hub.docker.com/_/python) .

As described in #15, it also works within a conda env.

By the way, I'll add more information about the working environment to README.

[m-ueno]

I'm closing this issue because #16 add information about the working environment to README. If you still have trouble, please open a new issue.