Open ggosiang opened 4 years ago
It looks like the frontend widget is failing to load the listing of available dashboards. Could you open the developer tools in your browser and check for failing network requests?
I don't see any error in network request field, all items with status code 200.
But in Console tab, I found two errors,
Failed to load resource: the server responded with a status of 404 () /user/testuser/nvdashboard/index.json?1569981966001:1
Uncaught (in promise) SyntaxError: Unexpected token < in JSON at position 1 lab:1
Yeah this definitely seems like a failure to grab the list of dashboards
Uncaught (in promise) SyntaxError: Unexpected token < in JSON at position 1 lab:1
This suggests an HTML response has been returned as it starts <
, likely an error page from Jupyter.
Your install commands above look a little off (you say jupyterlab-bokeh-server
but it should be jupyterlab-nvdashboard
). Did you follow these instructions?
Oh yes! I used following commands to install it, and without any error during installation
pip install jupyterlab-nvdashboard
jupyter labextension install jupyterlab-nvdashboard
(updated in initial question)
I have just tried to reproduce in a new conda environment with the following steps and am unable to.
$ conda create -n tmp-nvdashboard-24 python=3.7 nodejs ipython jupyterlab
$ pip install jupyterlab-nvdashboard
$ jupyter labextension install jupyterlab-nvdashboard
$ jupyter lab
Could you share the response from the page that is getting a 404?
Ok great thanks. And are there any errors in your Jupyter logs?
Can you provide the log location that I need to check?
I'm afraid that will depend on your Jupyter Hub setup. I would recommend you refer to the documentation you followed to set it up, or if someone else did then check with them.
Also just reproduced a working installation using JH following this post up until the RAPIDS section and then running the two installation commands from here with sudo -E
. Just wanted to rule out that this was an incompatibility with JH.
ok, checked. also without error in jupyter log
so seems this is not an incompatibility issue with JH.
Any chance @ian-r-rose could take a quick look here and let me know if I'm missing something?
It looks to me like you have covered most of the things I can think of @jacobtomlinson.
@ggosiang can you make sure that the jupyter_server_proxy
serverextension is installed an enabled?
jupyter serverextension list
in a terminal.
yes, jupyter_server_proxy is installed and enabled. but I found an error about server extension of jupyterlab-nvdashboard not sure how to solve it?
$ jupyter serverextension list
config dir: /home/jovyan/.jupyter
jupyterlab_git enabled
- Validating...
jupyterlab_git OK
jupyterlab-nvdashboard enabled
- Validating...
Error loading server extension jupyterlab-nvdashboard
X is jupyterlab-nvdashboard importable?
config dir: /opt/conda/etc/jupyter
jupyter_server_proxy enabled
- Validating...
jupyter_server_proxy OK
jupyterlab enabled
- Validating...
jupyterlab 1.0.4 OK
nbdime enabled
- Validating...
nbdime 1.1.0 OK
nbresuse enabled
- Validating...
nbresuse OK
jupyter_tensorboard enabled
- Validating...
jupyter_tensorboard 0.1.10 OK
$ jupyter labextension list
JupyterLab v1.0.4
Known labextensions:
app dir: /opt/conda/share/jupyter/lab
@jupyter-widgets/jupyterlab-manager v1.0.2 enabled OK
@jupyterlab/git v0.8.2 enabled OK
jupyterlab-nvdashboard v0.1.11 enabled OK
jupyterlab_bokeh v1.0.0 enabled OK
jupyterlab_tensorboard v0.1.6 enabled OK
nbdime-jupyterlab v1.0.0 enabled OK
Looks like you may need to change jupyterlab-nvdashboard
to jupyterlab_nvdashboard
. That is to say, the hyphen does not make a valid python import, it needs to be an underscore. You many need to go into the /home/jovyan/.jupyter
config dir to do that manually.
I've made a change in /home/jovyan/.jupyter/jupyter_notebook_config.json
jupyterlab-nvdashboard
-> jupyterlab_nvdashboard
but still got error
jupyterlab_nvdashboard enabled
- Validating...
X is jupyterlab_nvdashboard importable?
Hmm, can you verify that it is, in fact, importable?
yes, seems i can import it
I received the following error:
500 GET /nvdashboard/index.json?1576415340199 (127.0.0.1): could not start nvdashboard in time
Thanks for raising that @aradhyamathur, this sounds slightly different so I'm going to make a new issue from your comment.
No error running:
pip install jupyterlab-nvdashboard
jupyter labextension install jupyterlab-nvdashboard
But I get a blank GPU Dashboard...
When I run:
$ jupyter serverextension list
and $ jupyter labextension list
I get:
When inspecting the chrome browser error console I see:
Failed to load resource: the http://a157821:8888/nvdashboard/index.json?1576801385961 server responded with a status of 404 (Not Found)
Running jupyterlab within a docker container on a DGX station.
Thanks @antw-cg. I would expect that the issue is because it is trying to connect on the container hostname http://a157821:8888
which will not be routable from your browser.
Could you let me know the url you are using to connect to your Jupyter Lab instance?
Looking further into this it appears the hostname is being provided by Jupyter Lab in the ServerConnection.makeSettings({}).baseUrl
object.
When running in a container this hostname is the container's internal hostname, not the hostname where it is being exposed.
@jacobtomlinson URL I'm connecting to in the browser is http://a157821:8888/lab.
Looking further into this it appears the hostname is being provided by Jupyter Lab in the ServerConnection.makeSettings({}).baseUrl object.
Is my best bet to change ServerConnection.makeSettings({}).baseUrl
to http://a157821:8888
in the source code? I tried finding the above file in site-packages/jupyterlab_nvdashboard/
, but that directory only had
__init__.py __pycache__ apps server.py utils.py
If it's any help, the tensorboard jupyterlab extension is running in the same container and is working fine.
Ok that's fine if you can route to it then that's not the problem.
It seems to not be able to find the extension URL. Are there any errors in the Jupyter Lab logs?
I've followed installation steps below to install into my JL,
pip install jupyterlab-nvdashboard jupyter labextension install jupyterlab-nvdashboard
but didn't get any content when running code in CPU/GPU, need help
jupyterlab 1.0.4 jupyterlab-nvdashboard 0.1.11 bokeh 1.0.4
I am facing the same problem. Can anyone tell me the solution?
I tried to uninstall the extension and reinstall it again but now it didn't show up on the extension bar.
My guess is we still lack sufficient information to diagnose the problem. Are you able to share the logs from JupyterLab? Perhaps these will have the needed clue 😉
Where can I look for the logs? I tried to look for the logs in jupyterlab folder but nothing is there. And on the terminal, I don't see any errors.
@mohak2312 I recommend checking the browser console.
I ran into this, updating my Dockerfile fixed it. I guess jupyter may not properly restart from a running container. This should apply to Docker or Kubernetes (KubeFlow in my example). Add this to your jupyter Dockerfile
RUN apt install nodejs npm -y &&\
pip install jupyterlab-nvdashboard &&\
jupyter labextension install jupyterlab-nvdashboard
Here is a complete sample https://github.com/rapidsai/jupyterlab-nvdashboard/pull/43
Me too. followed installation steps below but got blank.
pip install jupyterlab-nvdashboard jupyter labextension install jupyterlab-nvdashboard
below is my jupyterlab's log. I am using windows 10.
the nvdashboard didn't show up on the extension bar in jupyter.
what's the problem?
@neoragex2002 we do not currently support windows. See #21.
@neoragex2002 we do not currently support windows. See #21.
@_@ errrrr thanks for the information! I suggest adding this to the installation instructions....
Is there any official answer for running this in a container?
Pulling the nightly docker container from rapids https://rapids.ai/start.html using the following:
docker pull rapidsai/rapidsai-dev-nightly:0.13-cuda10.0-devel-ubuntu16.04-py3.7
docker run --gpus all --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 \
rapidsai/rapidsai-dev-nightly:0.13-cuda10.0-devel-ubuntu16.04-py3.7
And running it on a DGX station is resulting in a blank dashboard like reported above. The chrome error console shows a lost websocket connection
@arnoldrw what do you see in the Jupyter log?
@jacobtomlinson a lot of logs like this.
Side question - is the jupyter log location for the rapids container documented anywhere? I couldn't find any documentation and had to kill and restart the process to be able to grab a log.
[I 12:48:41.313 NotebookApp] Build is up to date
WARNING:bokeh.server.util:Host wildcard '*' will allow connections originating from multiple (or possibly all) hostnames or IPs. Use non-wildcard values to restrict access explicitly
[I 12:48:41.746 NotebookApp] Kernel started: afef1a24-061a-4c25-b60d-f4bc0079fc77
[I 12:48:42.296 NotebookApp] Client sent subprotocols: ['bokeh', 'eyJzZXNzaW9uX2lkIjogIkF3NDR4VU9FaEhhNm5aRnV6bkdMVVZFQmxZZEhwMEtJRGlVTEtsTU1TYkhuIiwgInNlc3Npb25fZXhwaXJ5IjogMTU4NTIyNzIyMSwgImhlYWRlcnMiOiB7Ikhvc3QiOiAibG9jYWxob3N0Ojg4ODgiLCAiQ29ubmVjdGlvbiI6ICJrZWVwLWFsaXZlIiwgIlVwZ3JhZGUtSW5zZWN1cmUtUmVxdWVzdHMiOiAiMSIsICJVc2VyLUFnZW50IjogIk1vemlsbGEvNS4wIChNYWNpbnRvc2g7IEludGVsIE1hYyBPUyBYIDEwXzE1XzIpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUwsIGxpa2UgR2Vja28pIENocm9tZS84MC4wLjM5ODcuMTQ5IFNhZmFyaS81MzcuMzYiLCAiU2VjLUZldGNoLURlc3QiOiAiaWZyYW1lIiwgIkFjY2VwdCI6ICJ0ZXh0L2h0bWwsYXBwbGljYXRpb24veGh0bWwreG1sLGFwcGxpY2F0aW9uL3htbDtxPTAuOSxpbWFnZS93ZWJwLGltYWdlL2FwbmcsKi8qO3E9MC44LGFwcGxpY2F0aW9uL3NpZ25lZC1leGNoYW5nZTt2PWIzO3E9MC45IiwgIlNlYy1GZXRjaC1TaXRlIjogInNhbWUtb3JpZ2luIiwgIlNlYy1GZXRjaC1Nb2RlIjogIm5hdmlnYXRlIiwgIkFjY2VwdC1FbmNvZGluZyI6ICJnemlwIiwgIkFjY2VwdC1MYW5ndWFnZSI6ICJlbi1VUyxlbjtxPTAuOSIsICJDb29raWUiOiAiX3hzcmY9Mnw4MjYwZGYzYnw5YjE3Yzc1NmU1NDA3OTkwODVlNDJlYzQ3M2I1NWYwZXwxNTgyNjU0MjQxOyB1c2VybmFtZS1sb2NhbGhvc3QtODg4OT1cIjJ8MTowfDEwOjE1ODM1Mjk1NTB8MjM6dXNlcm5hbWUtbG9jYWxob3N0LTg4ODl8NDQ6WVRreU1HRXpZV1psTWpJd05EVTNZbUk1WldObFltVTNaakppWXpoaFkyTT18N2M0MmI1YTEzMjY2OTA3ZDlkODE3ZTMxNmY3MjcwMjdlMGViMjU5ZGU5YjAwNDVlNGIxNTNhZDFmYTJjYzVhY1wiOyBfZ2E9R0ExLjEuMTA2OTIyMTU2LjE1ODQxMTE5NjM7IF9oMm9haV9zaWQ9XCIyfDE6MHwxMDoxNTg1MTExNTI3fDEwOl9oMm9haV9zaWR8NDQ6WldNMFl6YzRZMlk0TkRBeE5EWTNZbUkxT1Rsa05Ea3haVFF6TjJZd05XWT18OTIxYTY2YThiNTEzMDU0Y2ZkM2E3ZDYxNjc1ODRiNmU5ZWZiZTgxZDhlMDQ0N2FkOWY2NWZlNTg3NjcxZTE0NVwiOyB1c2VybmFtZS1sb2NhbGhvc3QtODg4OD1cIjJ8MTowfDEwOjE1ODUyMjY5MjB8MjM6dXNlcm5hbWUtbG9jYWxob3N0LTg4ODh8NDQ6TVRJeVlXUm1PR0UzTXpJNE5EYzFPRGxoT0dWaFpHTTNZV1l4TjJaaFlXUT18OTk5MDI3MmQ5YzkwYjIyMmNkZDZhZmQ4ZDY5MTlhOTJhMmRiZDJmMWVlMmYwZjFiYTgxM2NiYWE1YWEzYzI3YVwiIiwgIklmLU5vbmUtTWF0Y2giOiAiXCIzODliNGU0NTQyODI0MzU2NjkwNGI4YmJiNWEyODMwYjkwNWU3Mjk5XCIiLCAiWC1Gb3J3YXJkZWQtQ29udGV4dCI6ICIvbnZkYXNoYm9hcmQiLCAiWC1Qcm94eWNvbnRleHRwYXRoIjogIi9udmRhc2hib2FyZCJ9LCAiY29va2llcyI6IHsiX3hzcmYiOiAiMnw4MjYwZGYzYnw5YjE3Yzc1NmU1NDA3OTkwODVlNDJlYzQ3M2I1NWYwZXwxNTgyNjU0MjQxIiwgInVzZXJuYW1lLWxvY2FsaG9zdC04ODg5IjogIjJ8MTowfDEwOjE1ODM1Mjk1NTB8MjM6dXNlcm5hbWUtbG9jYWxob3N0LTg4ODl8NDQ6WVRreU1HRXpZV1psTWpJd05EVTNZbUk1WldObFltVTNaakppWXpoaFkyTT18N2M0MmI1YTEzMjY2OTA3ZDlkODE3ZTMxNmY3MjcwMjdlMGViMjU5ZGU5YjAwNDVlNGIxNTNhZDFmYTJjYzVhYyIsICJfZ2EiOiAiR0ExLjEuMTA2OTIyMTU2LjE1ODQxMTE5NjMiLCAiX2gyb2FpX3NpZCI6ICIyfDE6MHwxMDoxNTg1MTExNTI3fDEwOl9oMm9haV9zaWR8NDQ6WldNMFl6YzRZMlk0TkRBeE5EWTNZbUkxT1Rsa05Ea3haVFF6TjJZd05XWT18OTIxYTY2YThiNTEzMDU0Y2ZkM2E3ZDYxNjc1ODRiNmU5ZWZiZTgxZDhlMDQ0N2FkOWY2NWZlNTg3NjcxZTE0NSIsICJ1c2VybmFtZS1sb2NhbGhvc3QtODg4OCI6ICIyfDE6MHwxMDoxNTg1MjI2OTIwfDIzOnVzZXJuYW1lLWxvY2FsaG9zdC04ODg4fDQ0Ok1USXlZV1JtT0dFM016STRORGMxT0RsaE9HVmhaR00zWVdZeE4yWmhZV1E9fDk5OTAyNzJkOWM5MGIyMjJjZGQ2YWZkOGQ2OTE5YTkyYTJkYmQyZjFlZTJmMGYxYmE4MTNjYmFhNWFhM2MyN2EifX0']
[I 12:48:42.297 NotebookApp] Trying to establish websocket connection to ws://localhost:59097/GPU-Memory/ws
[I 12:48:42.301 NotebookApp] Websocket connection established to ws://localhost:59097/GPU-Memory/ws
ERROR:tornado.application:Uncaught exception GET /GPU-Memory/ws (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/GPU-Memory/ws', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
File "/opt/conda/envs/rapids/lib/python3.7/site-packages/tornado/websocket.py", line 956, in _accept_connection
open_result = handler.open(*handler.open_args, **handler.open_kwargs)
File "/opt/conda/envs/rapids/lib/python3.7/site-packages/bokeh/server/views/ws.py", line 123, in open
raise ProtocolError("Subprotocol header is not 'bokeh'")
bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'
That appears to be related to #49. Please update to the latest release.
Thanks @jacobtomlinson ! That resolved it.
Is there anywhere I should mention that issue to get it into the nightly container?
Probably over here https://github.com/rapidsai/build
Thanks! Done https://github.com/rapidsai/build/issues/67 .
The best solution for this problem is to restart your container that will fix your problem.
Hello, I have tried the same installation on Azure Machine Learning service and I got the same blank dashboard. Steps to re-create:
Guys, need help with this.
@be-shekhar are you able to share the log from Jupyter Lab?
@jacobtomlinson Where shall I look for logs? Any specific file name?
@be-shekhar I'm afraid you will need to refer to the Azure documentation to find out where they are stored.
Okay @jacobtomlinson. I'll try and find out. What would be the location of log file if I install it in a local Ubuntu system?
When starting Jupyter Lab on a local system you usually would run the jupyter lab
command from the CLI, and the log would be output to the terminal, not a file.
Hi @jacobtomlinson I am facing the same issue, as suggested when i run jupyter lab i see this.
[I 06:27:32.518 LabApp] [nb_conda_kernels] enabled, 2 kernels found [I 06:27:32.746 LabApp] The port 8888 is already in use, trying another port. [I 06:27:33.205 LabApp] JupyterLab extension loaded from /anaconda/envs/l_py36/lib/python3.6/site-packages/jupyterlab [I 06:27:33.205 LabApp] JupyterLab application directory is /anaconda/envs/_py36/share/jupyter/lab [I 06:27:33.555 LabApp] [Jupytext Server Extension] Deriving a JupytextContentsManager from LargeFileManager [I 06:27:33.557 LabApp] Serving notebooks from local directory: /mnt/batch/tasks/shared/LS_root/mounts/clusters/nikgpu/code [I 06:27:33.557 LabApp] Jupyter Notebook 6.1.3 is running at: [I 06:27:33.557 LabApp] http://localhost:8889/?token=30fba730b9afed23fb20b844e5fb371dbc370689c05bcb78 [I 06:27:33.557 LabApp] or http://127.0.0.1:8889/?token=30fba730b9afed23fb20b844e5fb371dbc370689c05bcb78 [I 06:27:33.557 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [W 06:27:33.562 LabApp] No web browser found: could not locate runnable browser. [C 06:27:33.562 LabApp]
To access the notebook, open this file in a browser:
file:///home/azureuser/.local/share/jupyter/runtime/nbserver-55622-open.html
Or copy and paste one of these URLs:
http://localhost:8889/?token=30fba730b9afed23fb20b844e5fb371dbc370689c05bcb78
or http://127.0.0.1:8889/?token=30fba730b9afed23fb20b844e5fb371dbc370689c05bcb78
any suggestions on log folders? I tried everywhere else but did not find the related logs so that i can share here. Any help is much appreciated.
Seems it's still happening with v0.6.0
After installing with
pip3 install jupyterlab_nvdashboard
I check the extension and I see it in JL3.0.7 but using a tab like GPU optimization returns a blank page.
>>> jupyter labextension list
JupyterLab v3.0.7
/usr/local/share/jupyter/labextensions
jupyterlab-nvdashboard v0.6.0 enabled OK (python, jupyterlab_nvdashboard)
nbdime-jupyterlab v2.1.0 enabled OK
@jupyterlab/server-proxy v3.0.2 enabled OK
@jupyter-widgets/jupyterlab-manager v3.0.0 enabled OK (python, jupyterlab_widgets)
@ryantam626/jupyterlab_code_formatter v1.4.10 enabled OK (python, jupyterlab-code-formatter)
Other labextensions (built into JupyterLab)
app dir: /usr/local/share/jupyter/lab
[root@3d46a5c0d992 jupyter]# jupyter server extension list
Config dir: /usr/local/etc/jupyter
jupyter_server_proxy enabled
- Validating jupyter_server_proxy...
jupyter_server_proxy OK
jupyter_server_mathjax enabled
- Validating jupyter_server_mathjax...
jupyter_server_mathjax OK
jupyterlab enabled
- Validating jupyterlab...
jupyterlab 3.0.7 OK
jupyterlab_code_formatter enabled
- Validating jupyterlab_code_formatter...
jupyterlab_code_formatter 1.4.10 OK
jupyterlab_nvdashboard enabled
- Validating jupyterlab_nvdashboard...
X validation failed
nbclassic enabled
- Validating nbclassic...
nbclassic OK
nbdime enabled
- Validating nbdime...
nbdime 3.1.0 OK
Config dir: /usr/local/etc/jupyter
jupyter_server_proxy enabled
- Validating jupyter_server_proxy...
jupyter_server_proxy OK
jupyter_server_mathjax enabled
- Validating jupyter_server_mathjax...
jupyter_server_mathjax OK
jupyterlab enabled
- Validating jupyterlab...
jupyterlab 3.0.7 OK
jupyterlab_code_formatter enabled
- Validating jupyterlab_code_formatter...
jupyterlab_code_formatter 1.4.10 OK
jupyterlab_nvdashboard enabled
- Validating jupyterlab_nvdashboard...
X validation failed
nbclassic enabled
- Validating nbclassic...
nbclassic OK
nbdime enabled
- Validating nbdime...
nbdime 3.1.0 OK
Are there any extra steps for installation/configuration? Is this failed validation something to look further into?
Thanks for sharing further info. It would also be helpful if you could share your OS and version, Python version, etc.
When I run jupyter server extension list
I get
jupyterlab_nvdashboard enabled
- Validating jupyterlab_nvdashboard...
jupyterlab_nvdashboard OK
So it definitely sounds like something to investigate on your setup.
I'm installing Jupyterlab with extensions as a docker container to run inside Jupyterhub.
Using CentOS 7 Python 3.8.9
and the following dockerfile:
FROM my-centos7-python3.8.9-base
RUN wget https://github.com/krallin/tini/releases/download/v0.19.0/tini -O /usr/bin/tini
RUN chmod +x /usr/bin/tini
RUN pip3 install --no-cache-dir --upgrade pip==20.2.4 && pip3 --no-cache-dir install wheel
RUN pip3 install --no-cache-dir \
notebook \
ipywidgets \
tornado \
make \
bash_kernel \
pypki2 \
nbdime \
isort \
black \
jupyterlab_code_formatter \
jupyterlab_execute_time \
jupyterlab-nvdashboard==0.6.0 \
jupyterhub \
jupyterlab==3.0.7
RUN curl -sL https://rpm.nodesource.com/setup_lts.x | sudo -E bash - && yum -y install nodejs && yum clean all && rpm --rebuilddb && rm -rf /var/cache/yum
### Activate jupyterlab extensions ###
RUN nbdime extensions --enable \
&& jupyter labextension install --clean @ryantam626/jupyterlab_code_formatter \
&& jupyter serverextension enable --py jupyterlab_code_formatter --sys-prefix
COPY extensions/tracker.jupyterlab-settings $HOME/.jupyter/lab/user-settings/@jupyterlab/notebook-extension/tracker.jupyterlab-settings
RUN pip3 install --no-cache-dir jupyterlab-snippets \
&& jupyter lab build && npm update
# set startpoints
EXPOSE 8888
ENTRYPOINT ["tini", "-g", "--"]
CMD ["start-notebook.sh"]
I've followed installation steps below to install into my JL,
but didn't get any content when running code in CPU/GPU, need help