Can't use NVDashboard in the RAPIDS 23.04 runtime containers or in 23.04 conda environment with unpinned Jupyterlab

[beckernick]

When I run the RAPIDS 23.04 runtime containers locally, the NVDashboard Jupyter widget shows a blank screen. I'd like to be able to use Jupyterlab-NVDashboard in the 23.04 runtime containers.

docker run --gpus all --rm -it -p 8888:8888 -p 8797:8787 -p 8796:8786 nvcr.io/nvidia/rapidsai/rapidsai-core:23.04-cuda11.8-runtime-ubuntu22.04-py3.8

(rapids) root@d8935efd8303:/rapids/notebooks# conda list | grep jupyter  
jupyter-packaging         0.7.12             pyhd8ed1ab_0    conda-forge
jupyter-server-proxy      3.2.2              pyhd8ed1ab_0    conda-forge
jupyter_client            8.1.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.3.0            py38h578d9bd_0    conda-forge
jupyter_events            0.6.3              pyhd8ed1ab_0    conda-forge
jupyter_server            2.5.0              pyhd8ed1ab_0    conda-forge
jupyter_server_fileid     0.9.0              pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
jupyter_server_ydoc       0.8.0              pyhd8ed1ab_0    conda-forge
jupyter_ydoc              0.2.3              pyhd8ed1ab_0    conda-forge
jupyterlab                3.6.3              pyhd8ed1ab_0    conda-forge
jupyterlab-favorites      3.1.1              pyhd8ed1ab_0    conda-forge
jupyterlab-nvdashboard    0.8.0           py_230412_g8ed1064_0    rapidsai
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.22.0             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.7              pyhd8ed1ab_0    conda-forge

When I create a 23.04 environment that includes Jupyterlab (unpinned) and install NVDashboard (unpinned) via conda, the solver pulls in jupyterlab-nvdashboard v0.5.0 from the rapidsai channel rather than the current v0.8. It also pulls in Jupyterlab 4.0, which perhaps is related. In this environment, NVDashboard doesn't show up in the Jupyterlab left panel as an extension option. If I got to the explicit endpoint at http://localhost:8889/GPU-Utilization, I get a 404 error.

mamba create -n rapids-23.04-jupyter-unpinned -c rapidsai -c nvidia -c conda-forge rapids=23.04 python=3.10 cudatoolkit=11.8 jupyterlab jupyterlab-nvdashboard

If I create the same environment but pin Jupyterlab to v3, the solver pulls in Jupyterlab v3.6.3 and nvdashboard v0.8, with the same build hashes as the packages in the 23.04 runtime container, but different versions and build hashes for other Jupyter-related packages. In this bare metal conda environment, I'm able to use NVDashboard in Jupyterlab.

(rapids-23.04) nicholasb@nicholasb-HP-Z8-G4-Workstation:~$ conda list | grep jupy
jupyter-core              5.3.0                    pypi_0    pypi
jupyter-server-proxy      4.0.0              pyhd8ed1ab_0    conda-forge
jupyter_client            8.2.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.3.0           py310hff52083_0    conda-forge
jupyter_events            0.6.3              pyhd8ed1ab_0    conda-forge
jupyter_server            2.5.0              pyhd8ed1ab_0    conda-forge
jupyter_server_fileid     0.9.0              pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
jupyter_server_ydoc       0.8.0              pyhd8ed1ab_0    conda-forge
jupyter_ydoc              0.2.3              pyhd8ed1ab_0    conda-forge
jupyterlab                3.6.3              pyhd8ed1ab_0    conda-forge
jupyterlab-nvdashboard    0.8.0           py_230412_g8ed1064_0    rapidsai
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.22.1             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.7              pyhd8ed1ab_1    conda-forge

In the container, bokeh v2.4.3 comes from conda-forge. In the raw environment, it comes from PyPI.

Happy to provide full container/raw environment dependency diffs, if useful.

[jacobtomlinson]

If I create the same environment but pin Jupyterlab to v3, the solver pulls in Jupyterlab v3.6.3 and nvdashboard v0.8, with the same build hashes as the packages in the 23.04 runtime container, but different versions and build hashes for other Jupyter-related packages. In this bare metal conda environment, I'm able to use NVDashboard in Jupyterlab.

So just to be clear when you install this manually with JupyterLab pinned to v3 does the dashboard work? Or is it blank link in the RAPIDS container?

[beckernick]

If I create the same environment but pin Jupyterlab to v3, the solver pulls in Jupyterlab v3.6.3 and nvdashboard v0.8, with the same build hashes as the packages in the 23.04 runtime container, but different versions and build hashes for other Jupyter-related packages. In this bare metal conda environment, I'm able to use NVDashboard in Jupyterlab.

So just to be clear when you install this manually with JupyterLab pinned to v3 does the dashboard work? Or is it blank link in the RAPIDS container?

Yes, with this setup it works.

mamba create -n rapids-23.04 -c rapidsai -c nvidia -c conda-forge rapids=23.04 python=3.10 cudatoolkit=11.8 jupyterlab=3 jupyterlab-nvdashboard

[beckernick]

Perhaps relevant. If I remove the conda installed jupyterlab-nvdashboard and then install with pip, things appear to work:

docker run --gpus all --rm -it -p 8888:8888 -p 8797:8787 -p 8796:8786 nvcr.io/nvidia/rapidsai/rapidsai-core:23.04-cuda11.8-runtime-ubuntu22.04-py3.8
mamba remove jupyterlab-nvdashboard
pip install jupyterlab-nvdashboard
jupyter-lab

[jakirkham]

Out of curiosity is this still the case when using the conda-forge package? If not, maybe there is some difference between the two that is worth investigating

[jakirkham]

Out of curiosity is this still the case when using the conda-forge package? If not, maybe there is some difference between the two that is worth investigating

^ @beckernick curious if you had a chance to try the suggestion above?