search

yWorks / yfiles-jupyter-graphs

The home of the Jupyter notebook graph visualization widget powered by yFiles for HTML
https://www.yworks.com/products/yfiles-graphs-for-jupyter
Other
162 stars 15 forks source link

Plugin is not working with Sagemaker #52

Closed luizbergertrustly closed 5 months ago

luizbergertrustly commented 5 months ago

Describe the bug By deploying a notebook [Amazon Linux 2, Jupyter Lab 3 (notebook-al2-v2)] on Sagemaker, we want have the GraphWidget working on Jupyter Lab.

We are adding screenshots to showcase the problem. Restarting kernel does not change this behavior. We have also tried some setup on notebook start, no success.

To Reproduce Check on screenshots.

Expected behavior Have GraphWidget working on Sagemaker Jupyter Notebook.

Screenshots Creating the instance and launching it

image

Notebook is fresh, we will create a new notebook file

image

Available plugin

image

Installed lib, and now it's on showing on labextension list

image

Full error

image

Desktop (please complete the following information):

Additional context

We also have tried creating custom kernels, with many different library versionings, no success also. We had a similar problem on kubeflow, but since we were able to create the image that is going to be used, we have managed to find a good versioning to have it working there. Same versioning does not work on Sagemaker.

On kubeflow, we had this configuration for it to work, on Python 3.9.16:

RUN python3 -m pip uninstall -y \
    traitlets
# Downgrading was necessary, found it somewhere when troubleshooting
RUN python3 -m pip install --no-cache-dir \
    traitlets==5.9.0
# Libs related to Neo4j and Graph visualization
RUN python3 -m pip install --no-cache-dir \
    ipywidgets==8.1.1 \
    jupyterlab-widgets==3.0.9 \
    yfiles_jupyter_graphs==1.4.8

EDIT:

Adding versioning of the Sagemaker notebook:

IPython          : 8.22.2
ipykernel        : 6.29.3
ipywidgets       : 8.1.2
jupyter_client   : 8.6.1
jupyter_core     : 5.7.2
jupyter_server   : 2.13.0
jupyterlab       : 4.1.5
nbclient         : 0.10.0
nbconvert        : 7.16.3
nbformat         : 5.10.3
notebook         : 7.1.2
qtconsole        : 5.4.4
traitlets        : 5.14.2
yGuy commented 5 months ago

This may sound like poor advice, but did you actually try to simply reload the page in the browser? When a new widget gets installed on the kernel that uses JavaScript, the default jupyter notebook implementation does not manage to update the state of the widget registry in the browser. So restarting the kernel may not only be not enough, but actually the wrong tip. It only updates the data structures in the kernel but still doesn't update the widget registry.

So could you simply reload the browser and see if that helps?

I only ever saw this problem when installing the widget in the kernel and using it directly in the same browser session. This doesn't happen with Google Colab, e.g., because it dynamically fetches the widget from a CDN rather than loading it from the local kernel via the local registry.

I think we should copy that "important" bit from here: https://stackoverflow.com/a/60059786/351836

This is what phind.com had to say about this:

Yes, it is correct that after installing Jupyter widgets, you may need to reload the browser to make them work and appear in the widget registry. This is because the widgets rely on JavaScript libraries that are loaded by the browser. When you install or update these libraries, the browser needs to refresh its cache to recognize the new or updated widgets. This process ensures that the widgets are properly integrated into the Jupyter environment and can be interacted with as intended.

luizbergertrustly commented 5 months ago

Hey, @yGuy .

Thanks for reaching out. I have tried a different browser (Safari), also refreshing and force refreshing the page, but no success.

Also, under the Manager Extension tab, i have installed the plugin from there, no success, too. :(

image
yGuy commented 5 months ago

:-( - I don't know enough about how Sagemaker could be different from a regular jupyter notebook server.

Just yesterday I used this super simple docker image on my local machine:

FROM jupyter/scipy-notebook
RUN pip install yfiles-jupyter-graphs

With (what I think) is the most recent versions of the base image and our plugin and it still worked flawlessly for me.

This results in this configuration:

Python 3.11.6

JupyterLab v4.0.7
/opt/conda/share/jupyter/labextensions
        jupyterlab_pygments v0.2.2 enabled  X (python, jupyterlab_pygments)
        yfiles-jupyter-graphs v1.6.2 enabled OK (python, yfiles_jupyter_graphs)
        nbdime-jupyterlab v2.2.0 enabled  X
        jupyter-matplotlib v0.11.3 enabled OK
        @jupyter-notebook/lab-extension v7.0.6 enabled OK
        @jupyter-widgets/jupyterlab-manager v5.0.9 enabled OK (python, jupyterlab_widgets)
        @jupyterlab/git v0.41.0 enabled  X (python, jupyterlab-git)

   The following extensions are outdated:
        jupyterlab_pygments
        nbdime-jupyterlab
        @jupyterlab/git

   Consider checking if an update is available for these packages.

Other labextensions (built into JupyterLab)
   app dir: /opt/conda/share/jupyter/lab

It seems the docker base image is not up to date and your stack is slightly newer.

If you look at the developer console, is there anything suspicious? And if there isn't: Do you think you could provide me or my team with temporary access to such an installation.

yGuy commented 5 months ago

Maybe this comment or thread helps? https://github.com/matplotlib/ipympl/issues/460#issuecomment-1419304152

luizbergertrustly commented 5 months ago

I won't be able to share these installations, but it's the default environments of Sagemaker Notebooks, just some clicks to have it up and running.

I tried some procedures on this link, couldn't have it working.

There is other possible image, Amazon Linux 2, Jupyter Lab 1(notebook-al2-v1), which the Widget object can be created, but it simply does not render anything.

image
fskpf commented 5 months ago

A similar issue was discussed here https://github.com/jupyter-widgets/ipywidgets/issues/2220 with different solutions, all related to the setup of the environment.

Here as well https://forums.fast.ai/t/ipywidgets-not-displaying-on-aws-sagemaker-fastai/85039

And another discussion here: https://stackoverflow.com/questions/36351109/ipython-notebook-ipywidgets-does-not-show Maybe something like this or that does the trick.

Seems like ipywidgets + sagemaker is a tricky setup.

luizbergertrustly commented 5 months ago

Back here with good news.

For some reason, trying to do this configuration directly from the notebook in one of the available kernels is not the way.

A teammate here tried some other configuration, and at the end, we managed to have it working.

By having these commands here on notebook start (via lifecycle policy, it's one of the sagemaker resources):

/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin/python3 -m pip install yfiles_jupyter_graphs
/home/ec2-user/anaconda3/envs/python3/bin/python3 -m pip install yfiles_jupyter_graphs

The lib is installed both in the system default environment, and on the kernel that we are going to use on the notebook.

I assume that trying to do this directly from the notebook, was not properly installing the library on the desired environment. Because it seems that the jupyter application runs on top of the jupyter configuration of the system env. Don't know if it makes sense, but resolution is pretty simple (after spending some days and with help from others :D).

Anyways, thanks for your help @yGuy and @fskpf!