jupyterlmod does not work with Jupyter Lab

[dr-br]

~No Software is listed in the "Loaded Modules" or "Available Modules" sections.~ ~When I load modules in notebook (version 1.9.0, see #25 ), they are not displayed neither.~

~I tried different Jupyter {lab, notebook, ...} versions and many jupyterlmod versions and some jupyterlab-lmod@XXX versions. No success.~

The most recent version of jupyterlmod which works on our cluster is: jupyterlmod==1.9.0 (pip) jupyterlab-lmod@0.7.0 (labextension)

jupyter --version
jupyter core     : 4.6.3
jupyter-notebook : 6.1.4
qtconsole        : 4.7.7
ipython          : 7.16.1
ipykernel        : 5.3.4
jupyter client   : 6.1.7
jupyter lab      : 2.2.8
nbconvert        : 6.0.5
ipywidgets       : 7.5.1
nbformat         : 5.0.7
traitlets        : 4.3.3

pip freeze|grep jupyter
jupyter==1.0.0
jupyter-client==6.1.7
jupyter-console==6.2.0
jupyter-core==4.6.3
jupyter-server-proxy==1.5.0
jupyterlab==2.2.8
jupyterlab-pygments==0.1.1
jupyterlab-server==1.2.0
jupyterlmod==1.9.0

jupyter labextension list
JupyterLab v2.2.8
Known labextensions:
   app dir: /pfs/data5/home/kit/scc/ej4555/lmodtestenv/lmodtestenv/share/jupyter/lab
        jupyterlab-lmod v0.8.0  enabled  OK

module --version

Modules based on Lua: Version 8.3.15  2020-06-05 12:32 -05:00
    by Robert McLay mclay@tacc.utexas.edu

It would be fantastic, if the newest versions of jupyterlmod (python and npm package) would work on our cluster. Thanks!

[dr-br]

Would you please provide any hint, how to debug in order to give you useful information? I updated all software, the current versions are now:

jupyter --version
jupyter core     : 4.7.0
jupyter-notebook : 6.1.5
qtconsole        : 5.0.0
ipython          : 7.16.1
ipykernel        : 5.3.4
jupyter client   : 6.1.7
jupyter lab      : 2.2.9
nbconvert        : 6.0.7
ipywidgets       : 7.5.1
nbformat         : 5.0.8
traitlets        : 4.3.3

pip freeze|grep jupyter
jupyter-client==6.1.7
jupyter-core==4.7.0
jupyter-telemetry==0.1.0
jupyterhub==1.2.1
jupyterlab==2.2.9
jupyterlab-pygments==0.1.2
jupyterlab-server==1.2.0
jupyterlmod==2.0.1

jupyter labextension list
JupyterLab v2.2.9
Known labextensions:
   app dir: /pfs/data4/software_uc2/bwhpc/common/devel/jupyter_base/share/jupyter/lab
        jupyterlab-lmod v0.8.0  enabled  OK

module --version

Modules based on Lua: Version 8.3.15  2020-06-05 12:32 -05:00
    by Robert McLay mclay@tacc.utexas.edu

node --version
v10.21.0

It still does not work. Neither classic notebook nor JupyterLab. The site FQDN/user/myname/lmod shows: [] The old combination jupyterlmod==1.9.0 (pip) jupyterlab-lmod@0.7.0 (labextension) works.

[dr-br]

Maybe Browser output helps:

[cmd-ntrf]

The screenshot is a good start.

Can you expand the post request that returns a 400 error?

[dr-br]

which one? There are several tabs. Haders, Cookies, ...

[cmd-ntrf]

I meant this :

[dr-br]

[dr-br]

any other tab which might be interesting?

[dr-br]

maybe chromium is more helpful

[cmd-ntrf]

Can you try to locate the JSON that was posted by the request?

The 400 comes from this part of the jupyterlmod handler: https://github.com/cmd-ntrf/jupyter-lmod/blob/master/jupyterlmod/handler.py#L38

The posted payload does not match what is expected, so the handler returns a 400. The javascript part of jupyter-lmod is what is posting JSON data to the jupyter-lmod handler, so the format should match, but we would know for sure by looking at the json sent to the handler by your browser.

[dr-br]

Can you try to locate the JSON that was posted by the request?

how?

[cmd-ntrf]

This will not provide the json, but can you click on this (lmod.js) :

And provide a screenshot of the lines surrounding line 34?

[dr-br]

[cmd-ntrf]

The jupyter notebook logs should include the 400 error and tell us which one of the two 400 errors it is (either modules is absent from the json, or modules is not a list).

To retrieve the json from Chrome, there are instructions on stackoverflow: https://stackoverflow.com/questions/15603561/how-can-i-debug-a-http-post-in-chrome

[dr-br]

like i already wrote: module contains []

[cmd-ntrf]

modules is the list of available modules, not the json sent with the post.

That said, it should not be empty. If it is empty, it means the handler or the communication with lmod is not working properly. I don't know why at this point. I will have to look at what change in that regard between 1.9 and 2.0.

[dr-br]

[dr-br]

"Remove jquery from jupyterlab requirements" on July, 2 Could the first error (tooltip is not a function) be associated to this?

[cmd-ntrf]

The problem appears to come from the lmod Python module that does not return the list of available modules.

Since version 2.0 of jupyter-lmod, the parsing of lmod output is done with regular expressions : https://github.com/cmd-ntrf/jupyter-lmod/blob/master/lmod/__init__.py#L15

It is possible that your version of lmod output mismatch the regular expressions and jupyter-lmod fails in parsing lmod output. To test this, install jupyterlmod in a virtual environment on your cluster, and in an ipython shell execute the following commands:

%autoawait
import lmod
await lmod.avail()

It should return the list of available modules. If this list is empty, there is a problem with the regex or lmod output. If it is not empty, the problem lies elsewhere and I'll keep digging.

[dr-br]

Hi, nice debug hint!

python -m venv testlmod
source testlmod/bin/activate
pip install --upgrade pip
pip install git+https://github.com/cmd-ntrf/jupyter-lmod
ipython

Python 3.6.8 (default, Dec  5 2019, 15:45:45) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: %autoawait
IPython autoawait is `on`, and set to use `asyncio`
In [2]: import lmod
In [3]: await lmod.avail()
Out[3]: []
In [4]: 

Is there a way to debug further? Strace doesn't seem to give meaningful insight...

Maybe a sample output of module avail gives you any hint:

module avail

------------------------------------------- /opt/bwhpc/kit/modulefiles -------------------------------------------
   cae/abaqus/2020         cae/ansys/2020R2         cae/starccm+/2020.1        math/mathematica/12.1
   cae/ansys/19.2          cae/cst/2020             cae/starccm+/2020.2 (D)
   cae/ansys/2019R3        cae/lsdyna/11.1.0        cae/starcd/2019.1.2
   cae/ansys/2020R1 (D)    cae/lsdyna/931    (D)    chem/vasp/5.4.4.pl2

--------------------------------------- /opt/bwhpc/common/modulefiles/Core ---------------------------------------
   bio/freesurfer/6.0.0
   bio/fsl/5.0.9
   bio/fsl/6.0.1                                                  (D)
…

or

/opt/lmod/lmod/libexec/lmod python --terse avail
/opt/bwhpc/kit/modulefiles:
cae/abaqus/
cae/abaqus/2020
cae/ansys/
cae/ansys/19.2
cae/ansys/2019R3
cae/ansys/2020R1
cae/ansys/2020R2
cae/cst/
…

[dr-br]

Hi, we found the error: Our modules are arranged with depth 3 instead of 2. The original regex MODULE_REGEX assumes a depth of 2. So devel/myModule1.2.3 is found, devel/myModule/version1.2.3 is not. The regex below handles a depth of 2 and 3. Furthermore, we (actually, my colleague who knows regex) think, that …A-Z… is more appropriate than …A-z…

MODULE_REGEX = re.compile(r"^([a-zA-Z0-9-_+.]+[/][a-zA-Z0-9-_+.]+|[a-zA-Z0-9-_+.]+[/][a-zA-Z0-9-_+.]+[/][a-zA-Z0-9-_+.]+)$", re.M)

We do not propose any modification for MODULE_REGEX_NO_HIDDEN because we didn't get its meaning.

Sorry, this is unavoidable: https://xkcd.com/208/

Thank you very much @cmd-ntrf for your help!

[cmd-ntrf]

I am glad we could find the issue.

The original regex can be simplified to work with any hierarchy depth:

^[a-zA-Z0-9-_+.\/]{1,}[^\/:]$

You can try it here : https://regex101.com/r/003wfS/1

MODULE_REGEX_NO_HIDDEN, is the mostly same regex, but its purpose is to remove the modules that are hidden in lmod. Hidden modules have dot (.) at the beginning of their version number for example devel/gcc/.7.3.0 would be hidden.

This regex will remove the hidden module and work with any depth:

^[a-zA-Z0-9-_+.\/]{1,}\/[a-zA-Z0-9][a-zA-z0-9-_.]*[^\/:]$

I will test this thoroughly today and release jupyter-lmod 2.0.2.

Thank you for your patience and help in debugging this.

[cmd-ntrf]

Slightly improve version of MODULE_REGEX_NO_HIDDEN to allow modules with no slashes:

^([^.][a-zA-Z0-9-_+.]*|[a-zA-Z0-9-_+.\/]{1,}\/[a-zA-Z0-9][a-zA-z0-9-_.]*[^\/:])$

[cmd-ntrf]

@dr-br : can you try branch fix_regex and confirm it fixes your issue?

Thanks

[dr-br]

Works. Also resolves #25 ! Tnaks a lot! Samuel