search

CDAT / jupyter-vcdat

A Jupyter Lab extension based on vCDAT.
BSD 3-Clause "New" or "Revised" License
6 stars 5 forks source link

Issue using VCDAT at NERSC #164

Closed jasonltorchinsky closed 3 years ago

jasonltorchinsky commented 4 years ago

Describe the bug Jupyter lab does not load when the .override-jupyter file is in the HOME directory.

To Reproduce Steps to reproduce the behavior:

  1. Follow the steps provided at https://github.com/CDAT/jupyter-vcdat/wiki/Connecting#using-vcdat-at-nersc.
  2. When attempting to open a shared CPU node on Cori, the server will fail to spawn.

Expected behavior A Jupyter lab notebook to open with access to VCDAT/CDAT tools.

Screenshots This is the return message when attempting to open a shared CPU node on Cori. image

Desktop (please complete the following information):

Environment Information

`conda info`

``` active environment : base active env location : /global/common/cori_cle7/software/python/3.7-anaconda-2019.10 shell level : 1 user config file : /global/homes/j/jlturner/.condarc populated config files : conda version : 4.8.1 conda-build version : 3.18.9 python version : 3.7.4.final.0 virtual packages : __glibc=2.26 base environment : /global/common/cori_cle7/software/python/3.7-anaconda-2019.10 (read only) channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /global/common/cori_cle7/software/python/3.7-anaconda-2019.10/pkgs /global/homes/j/jlturner/.conda/pkgs envs directories : /global/homes/j/jlturner/.conda/envs /global/common/cori_cle7/software/python/3.7-anaconda-2019.10/envs platform : linux-64 user-agent : conda/4.8.1 requests/2.22.0 CPython/3.7.4 Linux/4.12.14-150.47-default sles/15 glibc/2.26 UID:GID : 80594:80594 netrc file : None offline mode : False ```

`conda list`

``` # packages in environment at /global/common/cori_cle7/software/python/3.7-anaconda-2019.10: # # Name Version Build Channel _ipyw_jlab_nb_ext_conf 0.1.0 py37_0 _libgcc_mutex 0.1 main alabaster 0.7.12 py37_0 anaconda 2019.10 py37_0 anaconda-client 1.7.2 py37_0 anaconda-navigator 1.9.7 py37_0 anaconda-project 0.8.3 py_0 asn1crypto 1.0.1 py37_0 astroid 2.3.1 py37_0 astropy 3.2.2 py37h7b6447c_0 atomicwrites 1.3.0 py37_1 attrs 19.2.0 py_0 babel 2.7.0 py_0 backcall 0.1.0 py37_0 backports 1.0 py_2 backports.functools_lru_cache 1.6.1 py_0 backports.os 0.1.1 py37_0 backports.shutil_get_terminal_size 1.0.0 py37_2 backports.tempfile 1.0 py_1 backports.weakref 1.0.post1 py_1 beautifulsoup4 4.8.0 py37_0 bitarray 1.0.1 py37h7b6447c_0 bkcharts 0.2 py37_0 blas 1.0 mkl bleach 3.1.0 py37_0 blosc 1.16.3 hd408876_0 bokeh 1.3.4 py37_0 boto 2.49.0 py37_0 bottleneck 1.2.1 py37h035aef0_1 buildtest 0.8.0 pypi_0 pypi bzip2 1.0.8 h7b6447c_0 ca-certificates 2019.8.28 0 cairo 1.14.12 h8948797_3 cartopy 0.17.0 py37hbb7e04d_1 certifi 2019.9.11 py37_0 cffi 1.12.3 py37h2e261b9_0 cftime 1.0.4.2 py37hdd07704_0 chardet 3.0.4 py37_1003 click 7.0 py37_0 cloudpickle 1.2.2 py_0 clyent 1.2.2 py37_1 colorama 0.4.1 py37_0 conda 4.8.1 py37_0 conda-build 3.18.9 py37_3 conda-env 2.6.0 1 conda-package-handling 1.6.0 py37h7b6447c_0 conda-verify 3.4.2 py_1 contextlib2 0.6.0 py_0 cryptography 2.7 py37h1ba5d50_0 curl 7.65.3 hbc83047_0 cycler 0.10.0 py37_0 cython 0.29.13 py37he6710b0_0 cytoolz 0.10.0 py37h7b6447c_0 dask 2.5.2 py_0 dask-core 2.5.2 py_0 dask-mpi 2.0.0 pypi_0 pypi dbus 1.13.6 h746ee38_0 decorator 4.4.0 py37_1 defusedxml 0.6.0 py_0 distributed 2.5.2 py_0 distro 1.4.0 pypi_0 pypi docutils 0.15.2 py37_0 entrypoints 0.3 py37_0 et_xmlfile 1.0.1 py37_0 expat 2.2.6 he6710b0_0 fastcache 1.1.0 py37h7b6447c_0 filelock 3.0.12 py_0 flask 1.1.1 py_0 fontconfig 2.13.0 h9420a91_0 freetype 2.9.1 h8a8886c_1 fribidi 1.0.5 h7b6447c_0 fsspec 0.5.2 py_0 future 0.18.2 py37_0 geos 3.7.1 he6710b0_0 get_terminal_size 1.0.0 haa9412d_0 gevent 1.4.0 py37h7b6447c_0 glib 2.56.2 hd408876_0 glob2 0.7 py_0 gmp 6.1.2 h6c8ec71_1 gmpy2 2.0.8 py37h10f8cd9_2 graphite2 1.3.13 h23475e2_0 greenlet 0.4.15 py37h7b6447c_0 gst-plugins-base 1.14.0 hbbd80ab_1 gstreamer 1.14.0 hb453b48_1 gunicorn 20.0.4 py37_0 h5py 2.9.0 py37h7918eee_0 harfbuzz 1.8.8 hffaf4a1_0 hdf4 4.2.13 h3ca952b_2 hdf5 1.10.4 hb1b8bf9_0 heapdict 1.0.1 py_0 html5lib 1.0.1 py37_0 icu 58.2 h9c2bf20_1 idna 2.8 py37_0 imageio 2.6.0 py37_0 imagesize 1.1.0 py37_0 importlib_metadata 0.23 py37_0 intel-openmp 2019.4 243 ipykernel 5.1.2 py37h39e3cac_0 ipympl 0.5.6 pypi_0 pypi ipyparallel 6.2.4 py37_0 ipython 7.8.0 py37h39e3cac_0 ipython_genutils 0.2.0 py37_0 ipywidgets 7.5.1 py_0 isort 4.3.21 py37_0 itsdangerous 1.1.0 py37_0 jbig 2.1 hdba287a_0 jdcal 1.4.1 py_0 jedi 0.15.1 py37_0 jeepney 0.4.1 py_0 jinja2 2.10.3 py_0 joblib 0.13.2 py37_0 jpeg 9b h024ee3a_2 json5 0.8.5 py_0 jsonschema 3.0.2 py37_0 jupyter 1.0.0 py37_7 jupyter_client 5.3.3 py37_1 jupyter_console 6.0.0 py37_0 jupyter_core 4.5.0 py_0 jupyterlab 1.1.4 pyhf63ae98_0 jupyterlab_server 1.0.6 py_0 keyring 18.0.0 py37_0 kiwisolver 1.1.0 py37he6710b0_0 krb5 1.16.1 h173b8e3_7 lazy-object-proxy 1.4.2 py37h7b6447c_0 libarchive 3.3.3 h5d8350f_5 libcurl 7.65.3 h20c2e04_0 libedit 3.1.20181209 hc058e9b_0 libffi 3.2.1 hd88cf55_4 libgcc-ng 9.1.0 hdf63c60_0 libgfortran-ng 7.3.0 hdf63c60_0 liblief 0.9.0 h7725739_2 libnetcdf 4.6.1 h11d0813_2 libpng 1.6.37 hbc83047_0 libpq 11.2 h20c2e04_0 libprotobuf 3.6.0 hdbcaa40_0 libsodium 1.0.16 h1bed415_0 libssh2 1.8.2 h1ba5d50_0 libstdcxx-ng 9.1.0 hdf63c60_0 libtiff 4.0.10 h2733197_2 libtool 2.4.6 h7b6447c_5 libuuid 1.0.3 h1bed415_2 libxcb 1.13 h1bed415_1 libxml2 2.9.9 hea5a465_1 libxslt 1.1.33 h7d1a2b0_0 llvmlite 0.29.0 py37hd408876_0 locket 0.2.0 py37_1 lxml 4.4.1 py37hefd8a0e_0 lz4-c 1.8.1.2 h14c3975_0 lzo 2.10 h49e0be7_2 markupsafe 1.1.1 py37h7b6447c_0 matplotlib 3.1.1 py37h5429711_0 mccabe 0.6.1 py37_1 mistune 0.8.4 py37h7b6447c_0 mkl 2019.4 243 mkl-service 2.3.0 py37he904b0f_0 mkl_fft 1.0.14 py37ha843d7b_0 mkl_random 1.1.0 py37hd6b4f25_0 mock 3.0.5 py37_0 more-itertools 7.2.0 py37_0 mpc 1.1.0 h10f8cd9_1 mpfr 4.0.1 hdf1c602_3 mpi4py 3.0.3 pypi_0 pypi mpmath 1.1.0 py37_0 msgpack-python 0.6.1 py37hfd86e86_1 multipledispatch 0.6.0 py37_0 mysql-connector-c 6.1.11 h597af5e_0 mysql-connector-python 8.0.18 py37h9c95fcb_1 navigator-updater 0.2.1 py37_0 nbconvert 5.6.0 py37_1 nbformat 4.4.0 py37_0 ncurses 6.1 he6710b0_1 netcdf4 1.4.2 py37h808af73_0 networkx 2.3 py_0 nltk 3.4.5 py37_0 nose 1.3.7 py37_2 notebook 6.0.1 py37_0 numba 0.45.1 py37h962f231_0 numexpr 2.7.0 py37h9e4a6bb_0 numpy 1.17.2 py37haad9e8e_0 numpy-base 1.17.2 py37hde5b4d6_0 numpydoc 0.9.1 py_0 olefile 0.46 py37_0 openpyxl 3.0.0 py_0 openssl 1.1.1d h7b6447c_2 owslib 0.18.0 py_0 packaging 19.2 py_0 pandas 0.25.1 py37he6710b0_0 pandoc 2.2.3.2 0 pandocfilters 1.4.2 py37_1 pango 1.42.4 h049681c_0 parso 0.5.1 py_0 partd 1.0.0 py_0 patchelf 0.9 he6710b0_3 path.py 12.0.1 py_0 pathlib2 2.3.5 py37_0 patsy 0.5.1 py37_0 pcre 8.43 he6710b0_0 pep8 1.7.1 py37_0 pexpect 4.7.0 py37_0 pickleshare 0.7.5 py37_0 pillow 6.2.0 py37h34e0f95_0 pip 19.2.3 py37_0 pixman 0.38.0 h7b6447c_0 pkginfo 1.5.0.1 py37_0 pluggy 0.13.0 py37_0 ply 3.11 py37_0 proj4 5.2.0 he6710b0_1 prometheus_client 0.7.1 py_0 prompt_toolkit 2.0.10 py_0 protobuf 3.6.0 py37hf484d3e_0 psutil 5.6.3 py37h7b6447c_0 psycopg2 2.8.4 py37h1ba5d50_0 ptyprocess 0.6.0 py37_0 py 1.8.0 py37_0 py-lief 0.9.0 py37h7725739_2 pycodestyle 2.5.0 py37_0 pycosat 0.6.3 py37h14c3975_0 pycparser 2.19 py37_0 pycrypto 2.6.1 py37h14c3975_9 pycurl 7.43.0.3 py37h1ba5d50_0 pyepsg 0.4.0 py37_0 pyflakes 2.1.1 py37_0 pygments 2.4.2 py_0 pykdtree 1.3.1 py37hdd07704_2 pylint 2.4.2 py37_0 pymongo 3.9.0 py37he6710b0_0 pyodbc 4.0.27 py37he6710b0_0 pyopenssl 19.0.0 py37_0 pyparsing 2.4.2 py_0 pyproj 1.9.6 py37h14380d9_0 pyqt 5.9.2 py37h05f1152_2 pyrsistent 0.15.4 py37h7b6447c_0 pyshp 2.1.0 py_0 pysocks 1.7.1 py37_0 pytables 3.5.2 py37h71ec239_1 pytest 5.2.1 py37_0 pytest-arraydiff 0.3 py37h39e3cac_0 pytest-astropy 0.5.0 py37_0 pytest-doctestplus 0.4.0 py_0 pytest-openfiles 0.4.0 py_0 pytest-remotedata 0.3.2 py37_0 python 3.7.4 h265db76_1 python-dateutil 2.8.0 py37_0 python-libarchive-c 2.8 py37_13 pytz 2019.3 py_0 pywavelets 1.0.3 py37hdd07704_1 pyyaml 5.1.2 py37h7b6447c_0 pyzmq 18.1.0 py37he6710b0_0 qt 5.9.7 h5867ecd_1 qtawesome 0.6.0 py_0 qtconsole 4.5.5 py_0 qtpy 1.9.0 py_0 readline 7.0 h7b6447c_5 requests 2.22.0 py37_0 ripgrep 0.10.0 hc07d326_0 rope 0.14.0 py_0 ruamel_yaml 0.15.46 py37h14c3975_0 scikit-image 0.15.0 py37he6710b0_0 scikit-learn 0.21.3 py37hd81dba3_0 scipy 1.3.1 py37h7c811a0_0 seaborn 0.9.0 py37_0 secretstorage 3.1.1 py37_0 send2trash 1.5.0 py37_0 setuptools 41.4.0 py37_0 shapely 1.6.4 py37h86c5351_0 simplegeneric 0.8.1 py37_2 singledispatch 3.4.0.3 py37_0 sip 4.19.8 py37hf484d3e_0 six 1.12.0 py37_0 snappy 1.1.7 hbae5bb6_3 snowballstemmer 2.0.0 py_0 sortedcollections 1.1.2 py37_0 sortedcontainers 2.1.0 py37_0 soupsieve 1.9.3 py37_0 sphinx 2.2.0 py_0 sphinxcontrib 1.0 py37_1 sphinxcontrib-applehelp 1.0.1 py_0 sphinxcontrib-devhelp 1.0.1 py_0 sphinxcontrib-htmlhelp 1.0.2 py_0 sphinxcontrib-jsmath 1.0.1 py_0 sphinxcontrib-qthelp 1.0.2 py_0 sphinxcontrib-serializinghtml 1.1.3 py_0 sphinxcontrib-websupport 1.1.2 py_0 spyder 3.3.6 py37_0 spyder-kernels 0.5.2 py37_0 sqlalchemy 1.3.9 py37h7b6447c_0 sqlite 3.30.0 h7b6447c_0 statsmodels 0.10.1 py37hdd07704_0 sympy 1.4 py37_0 tabulate 0.8.3 py37_0 tbb 2019.4 hfd86e86_0 tblib 1.4.0 py_0 terminado 0.8.2 py37_0 testpath 0.4.2 py37_0 tk 8.6.8 hbc83047_0 toolz 0.10.0 py_0 tornado 6.0.3 py37h7b6447c_0 tqdm 4.36.1 py_0 traitlets 4.3.3 py37_0 unicodecsv 0.14.1 py37_0 unixodbc 2.3.7 h14c3975_0 urllib3 1.24.2 py37_0 wcwidth 0.1.7 py37_0 webencodings 0.5.1 py37_1 werkzeug 0.16.0 py_0 wheel 0.33.6 py37_0 widgetsnbextension 3.5.1 py37_0 wrapt 1.11.2 py37h7b6447c_0 wurlitzer 1.0.3 py37_0 xlrd 1.2.0 py37_0 xlsxwriter 1.2.1 py_0 xlwt 1.3.0 py37_0 xz 5.2.4 h14c3975_4 yaml 0.1.7 had09818_2 zeromq 4.3.1 he6710b0_3 zict 1.0.0 py_0 zipp 0.6.0 py_0 zlib 1.2.11 h7b6447c_3 zstd 1.3.7 h0b5b093_0 ```

Additional context Jupyter lab opens just fine once the .override-jupyter file is deleted. According to the NERSC help desk, a relevant part of my .override-jupyter file is

2020-08-12T06:13:42-07:00 LAUNCH
Traceback (most recent call last):
File "/opt/conda/bin/jupyter-labhub", line 6, in <module>
from jupyterlab.labhubapp import main
ModuleNotFoundError: No module named 'jupyterlab.labhubapp'
2020-08-12T07:14:07-07:00 LAUNCH
shifter: unrecognized option '--jupyter-labhub'
Missing an argument!

Usage:
shifter [-h|--help] [-v|--verbose] [--image=<imageType>:<imageTag>]
[--entrypoint[=command]] [--workdir[=/path]]
[-E|--clearenv] [-e|--env=<var>=<value>] [--env-file=/env/file
[-V|--volume=/path/to/bind:/mnt/in/image[:<flags>[,...]][;...]]
[-m|--module=<modulename>[,...]]
[-- /command/to/exec/in/shifter [args...]]

which appears to suggest that the tutorial for writing the .override-jupyter file is out of date.

I also can't locate the /home/jovyan directory that the .override-jupyter file seems to mention.

downiec commented 4 years ago

@jasonlturner thanks for mentioning this. We'll look into the issue and update the documentation as needed!

sterlingbaldwin commented 4 years ago

I believe this is a NERSC infrastructure problem. I've seen this issue before on one of our clusters and it was because of a stuck node that needed to be rebooted.

jasonltorchinsky commented 4 years ago

Is there something that I can do to resolve this issue? Who can I contact about rebooting a node?

downiec commented 3 years ago

@jasonlturner I was finally able to get login access to Cori and troubleshoot the issue. After pulling in the latest VCDAT image with shifter, it looks like everything is working again. Please try it and let me know if VCDAT is loading for you now.