Support for AI Platform Notebooks

[gogasca]

AI Platform Notebooks are accessible via Inverse Proxy.

Inverse Proxy only support backend on JupyterLab port 8080. This plugin is launching a new web server in a random port, hence when we install it, and try to display GPU information just a blank screen is displayed, as Proxy cant route back the requests on this new web server.

Questions:

1. Can you support rendering the details inside JupyterLab without launching a new webserver?
2. If #1 is not possible, can you add support for a URL can be used which will be used to render the contents.
3. I'm able to start ngrok on this random port and access the local URL. I would like to be able to provide this URL to extension and when I click in extension content is rendered.

ngrok generates a public URL: https://9d4232d76b1b.ngrok.io/

Flow:

• Extension will allow me to configure a public URL: https://9d4232d76b1b.ngrok.io
• When I click on GPU-memory it will render: and it will just access URL + resource. Example: https://9d4232d76b1b.ngrok.io/GPU-Memory, etc.

[jacobtomlinson]

Thanks for raising this.

This extension uses the jupyter-server-proxy library to reverse proxy traffic via Jupyter Lab. So it shouldn't matter what port the extension is running on behind the scenes because everything is exposed via the Jupyter port.

If the dashboard is not displaying correctly this could be an issue with the way the URL is being constructed.

Could you please share a minimal reproducible example so we can check this out?

[gogasca]

Hi Jacob, thanks for responding,

These are the steps to reproduce:

1. Create an AI Platform Notebooks in Google Cloud.
2. Install NVIDIA plugin

Login via SSH to Notebooks and enter the following commands:

pip install jupyterlab-nvdashboard
jupyter labextension install jupyterlab-nvdashboard
jupyter lab build
service jupyter restart (or reboot)

3. Restart Jupyter
4. Access JupyterLab URL. (URL that Notebooks generate is something like this: https://XXXXXXXXXXX-dot-us-west1.notebooks.googleusercontent.com/lab) this URL provides access to VM and its behind an Inverse Proxy Browser <----> [Google Cloud Inverse Proxy] <------> [ Proxy Agent <-----> JupyterLab ]

• Google Cloud Inverse Proxy sits inside Google infrastructure
• Proxy Agent is a docker container running inside the Notebook
• 5. Access GPU Dashboards (Buttons are displayed but when I clicked on them I get a blank page)
  6. Logs

sudo journalctl -u jupyter.service --no-pager
Jan 26 17:36:31 nvidia-test bash[1315]: [I 17:36:31.415 LabApp] Trying to establish websocket connection to ws://localhost:40055/GPU-Utilization/ws
Jan 26 17:36:31 nvidia-test bash[1315]: [I 17:36:31.418 LabApp] Websocket connection established to ws://localhost:40055/GPU-Utilization/ws
Jan 26 17:36:31 nvidia-test bash[1315]: ERROR:tornado.application:Uncaught exception GET /GPU-Utilization/ws (::1)
Jan 26 17:36:31 nvidia-test bash[1315]: HTTPServerRequest(protocol='http', host='localhost:8080', method='GET', uri='/GPU-Utilization/ws', version='HTTP/1.1', remote_ip='::1')
Jan 26 17:36:31 nvidia-test bash[1315]: Traceback (most recent call last):
Jan 26 17:36:31 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/tornado/websocket.py", line 954, in _accept_connection
Jan 26 17:36:31 nvidia-test bash[1315]:     open_result = handler.open(*handler.open_args, **handler.open_kwargs)
Jan 26 17:36:31 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/tornado/web.py", line 3173, in wrapper
Jan 26 17:36:31 nvidia-test bash[1315]:     return method(self, *args, **kwargs)
Jan 26 17:36:31 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/bokeh/server/views/ws.py", line 125, in open
Jan 26 17:36:31 nvidia-test bash[1315]:     raise ProtocolError("Subprotocol header is not 'bokeh'")
Jan 26 17:36:31 nvidia-test bash[1315]: bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'
Jan 26 17:36:31 nvidia-test bash[1315]: [I 17:36:31.507 LabApp] Kernel started: 9ecf1159-3cde-4870-8865-46099f1c3bd6, name: python3
Jan 26 17:36:31 nvidia-test bash[1315]: [I 17:36:31.596 LabApp] Trying to establish websocket connection to ws://localhost:40055/GPU-Memory/ws
Jan 26 17:36:31 nvidia-test bash[1315]: [I 17:36:31.600 LabApp] Websocket connection established to ws://localhost:40055/GPU-Memory/ws
Jan 26 17:36:31 nvidia-test bash[1315]: ERROR:tornado.application:Uncaught exception GET /GPU-Memory/ws (::1)
Jan 26 17:36:31 nvidia-test bash[1315]: HTTPServerRequest(protocol='http', host='localhost:8080', method='GET', uri='/GPU-Memory/ws', version='HTTP/1.1', remote_ip='::1')
Jan 26 17:36:31 nvidia-test bash[1315]: Traceback (most recent call last):
Jan 26 17:36:31 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/tornado/websocket.py", line 954, in _accept_connection
Jan 26 17:36:31 nvidia-test bash[1315]:     open_result = handler.open(*handler.open_args, **handler.open_kwargs)
Jan 26 17:36:31 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/tornado/web.py", line 3173, in wrapper
Jan 26 17:36:31 nvidia-test bash[1315]:     return method(self, *args, **kwargs)
Jan 26 17:36:31 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/bokeh/server/views/ws.py", line 125, in open
Jan 26 17:36:31 nvidia-test bash[1315]:     raise ProtocolError("Subprotocol header is not 'bokeh'")
Jan 26 17:36:31 nvidia-test bash[1315]: bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'
Jan 26 17:36:35 nvidia-test bash[1315]: [I 17:36:35.497 LabApp] Trying to establish websocket connection to ws://localhost:40055/GPU-Utilization/ws
Jan 26 17:36:35 nvidia-test bash[1315]: [I 17:36:35.500 LabApp] Websocket connection established to ws://localhost:40055/GPU-Utilization/ws
Jan 26 17:36:35 nvidia-test bash[1315]: ERROR:tornado.application:Uncaught exception GET /GPU-Utilization/ws (::1)
Jan 26 17:36:35 nvidia-test bash[1315]: HTTPServerRequest(protocol='http', host='localhost:8080', method='GET', uri='/GPU-Utilization/ws', version='HTTP/1.1', remote_ip='::1')
Jan 26 17:36:35 nvidia-test bash[1315]: Traceback (most recent call last):
Jan 26 17:36:35 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/tornado/websocket.py", line 954, in _accept_connection
Jan 26 17:36:35 nvidia-test bash[1315]:     open_result = handler.open(*handler.open_args, **handler.open_kwargs)
Jan 26 17:36:35 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/tornado/web.py", line 3173, in wrapper
Jan 26 17:36:35 nvidia-test bash[1315]:     return method(self, *args, **kwargs)
Jan 26 17:36:35 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/bokeh/server/views/ws.py", line 125, in open
Jan 26 17:36:35 nvidia-test bash[1315]:     raise ProtocolError("Subprotocol header is not 'bokeh'")
Jan 26 17:36:35 nvidia-test bash[1315]: bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'
Jan 26 17:36:35 nvidia-test bash[1315]: [W 17:36:35.752 LabApp] 403 GET /metrics?1611682595548 (::1) 1.500000ms referer=https://6000049a22253f1c-dot-us-west1.notebooks.googleusercontent.com/lab?authuser=0
Jan 26 17:36:38 nvidia-test bash[1315]: [I 17:36:38.767 LabApp] Trying to establish websocket connection to ws://localhost:40055/GPU-Resources/ws
Jan 26 17:36:38 nvidia-test bash[1315]: BokehDeprecationWarning: 'legend' keyword is deprecated, use explicit 'legend_label', 'legend_field', or 'legend_group' keywords instead
Jan 26 17:36:38 nvidia-test bash[1315]: ERROR:tornado.application:Uncaught exception GET /GPU-Resources/ws (::1)
Jan 26 17:36:38 nvidia-test bash[1315]: HTTPServerRequest(protocol='http', host='localhost:8080', method='GET', uri='/GPU-Resources/ws', version='HTTP/1.1', remote_ip='::1')
Jan 26 17:36:38 nvidia-test bash[1315]: Traceback (most recent call last):
Jan 26 17:36:38 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/tornado/websocket.py", line 954, in _accept_connection
Jan 26 17:36:38 nvidia-test bash[1315]:     open_result = handler.open(*handler.open_args, **handler.open_kwargs)
Jan 26 17:36:38 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/tornado/web.py", line 3173, in wrapper
Jan 26 17:36:38 nvidia-test bash[1315]:     return method(self, *args, **kwargs)
Jan 26 17:36:38 nvidia-test bash[1315]:   File "/opt/conda/lib/python3.7/site-packages/bokeh/server/views/ws.py", line 125, in open
Jan 26 17:36:38 nvidia-test bash[1315]:     raise ProtocolError("Subprotocol header is not 'bokeh'")
Jan 26 17:36:38 nvidia-test bash[1315]: bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'
Jan 26 17:36:38 nvidia-test bash[1315]: [I 17:36:38.775 LabApp] Websocket connection established to ws://localhost:40055/GPU-Resources/ws

conda list

(base) jupyter@nvidia-test:~$ conda list
# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
abseil-cpp                20200923.3           h9c3ff4c_0    conda-forge
absl-py                   0.11.0                   pypi_0    pypi
adal                      1.2.5              pyh9f0ad1d_0    conda-forge
aiohttp                   3.7.3            py37h5e8e339_1    conda-forge
ansiwrap                  0.8.4                      py_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
argon2-cffi               20.1.0           py37h5e8e339_2    conda-forge
arrow                     0.17.0           py37h89c1867_1    conda-forge
arrow-cpp                 2.0.0           py37hc9ed16a_17_cpu    conda-forge
asn1crypto                1.4.0              pyh9f0ad1d_0    conda-forge
astropy                   4.2              py37h5e8e339_1    conda-forge
astunparse                1.6.3                    pypi_0    pypi
async-timeout             3.0.1                   py_1000    conda-forge
async_generator           1.10                       py_0    conda-forge
attrs                     20.3.0             pyhd3deb0d_0    conda-forge
aws-c-cal                 0.4.5                h54aeb68_4    conda-forge
aws-c-common              0.4.65               h7f98852_0    conda-forge
aws-c-event-stream        0.2.6                hbb68377_1    conda-forge
aws-c-io                  0.8.0                h650c6ac_0    conda-forge
aws-checksums             0.1.10               h650c6ac_2    conda-forge
aws-sdk-cpp               1.8.127              hf85971d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.1                      py_0    conda-forge
binaryornot               0.4.4                      py_1    conda-forge
black                     20.8b1                     py_1    conda-forge
bleach                    3.2.2              pyh44b312d_0    conda-forge
blinker                   1.4                        py_1    conda-forge
bokeh                     2.2.3                    pypi_0    pypi
bottleneck                1.3.2            py37h902c9e0_3    conda-forge
brotli                    1.0.9                h9c3ff4c_4    conda-forge
brotlipy                  0.7.0           py37h5e8e339_1001    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h36c2ea0_0    conda-forge
ca-certificates           2020.12.5            ha878542_0    conda-forge
cachetools                4.2.0                    pypi_0    pypi
caip-notebooks-serverextension 1.0.0                    pypi_0    pypi
cairo                     1.16.0            h7979940_1007    conda-forge
certifi                   2020.12.5        py37h89c1867_1    conda-forge
cffi                      1.14.4           py37hc58025e_1    conda-forge
chardet                   3.0.4           py37he5f6b98_1008    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
cloudpickle               1.3.0                    pypi_0    pypi
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
conda                     4.9.2            py37h89c1867_0    conda-forge
conda-package-handling    1.7.2            py37hb5d75c8_0    conda-forge
confuse                   1.4.0              pyhd3deb0d_0    conda-forge
cookiecutter              1.7.2              pyh9f0ad1d_0    conda-forge
cryptography              3.3.1            py37h7f0c10b_1    conda-forge
cycler                    0.10.0                     py_2    conda-forge
dataclasses               0.7                pyhb2cacf7_7    conda-forge
dbus                      1.13.6               hfdff14a_1    conda-forge
decorator                 4.4.2                      py_0    conda-forge
defusedxml                0.6.0                      py_0    conda-forge
dill                      0.3.3                    pypi_0    pypi
dlenv-base                1.0.20210121     py37he1a07ef_0    file:///tmp/conda-pkgs
dlenv-tf-2-4-gpu          1.0.20210121     py37h900f063_0    <unknown>
dm-tree                   0.1.5                    pypi_0    pypi
docker-py                 4.4.1            py37h89c1867_0    conda-forge
docker-pycreds            0.4.0                      py_0    conda-forge
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
expat                     2.2.9                he1b5a44_2    conda-forge
explainable-ai-sdk        1.1.0                    pypi_0    pypi
explainers                0.1                      pypi_0    pypi
ffmpeg                    4.3.1                hca11adc_2    conda-forge
flatbuffers               1.12                     pypi_0    pypi
fontconfig                2.13.1            hba837de_1004    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
fsspec                    0.8.5              pyhd8ed1ab_0    conda-forge
future                    0.18.2                   pypi_0    pypi
gast                      0.3.3                    pypi_0    pypi
gcsfs                     0.7.1                      py_0    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
gitdb                     4.0.5                      py_0    conda-forge
gitpython                 3.1.12             pyhd8ed1ab_0    conda-forge
glib                      2.66.4               hcd2ae1e_1    conda-forge
glog                      0.4.0                h49b9bf7_3    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
google-api-core           1.22.4             pyh9f0ad1d_0    conda-forge
google-api-core-grpcio-gcp 1.22.2               hc8dfbb8_0    conda-forge
google-api-python-client  1.12.8             pyhd3deb0d_0    conda-forge
google-auth               1.24.0             pyhd3deb0d_0    conda-forge
google-auth-httplib2      0.0.4              pyh9f0ad1d_0    conda-forge
google-auth-oauthlib      0.4.2              pyhd8ed1ab_0    conda-forge
google-cloud-bigquery     1.26.1                   pypi_0    pypi
google-cloud-bigquery-storage 1.0.0                    pypi_0    pypi
google-cloud-bigtable     1.4.0                    pypi_0    pypi
google-cloud-core         1.3.0                    pypi_0    pypi
google-cloud-dataproc     1.1.1                    pypi_0    pypi
google-cloud-datastore    1.12.0                   pypi_0    pypi
google-cloud-firestore    1.8.1                    pypi_0    pypi
google-cloud-kms          1.4.0                    pypi_0    pypi
google-cloud-language     1.3.0                    pypi_0    pypi
google-cloud-logging      1.15.1                   pypi_0    pypi
google-cloud-monitoring   1.1.0                    pypi_0    pypi
google-cloud-pubsub       1.7.0                    pypi_0    pypi
google-cloud-scheduler    1.3.0                    pypi_0    pypi
google-cloud-spanner      1.17.1                   pypi_0    pypi
google-cloud-speech       1.3.2                    pypi_0    pypi
google-cloud-storage      1.30.0                   pypi_0    pypi
google-cloud-tasks        1.5.0                    pypi_0    pypi
google-cloud-translate    2.0.2                    pypi_0    pypi
google-cloud-videointelligence 1.15.0                   pypi_0    pypi
google-cloud-vision       1.0.0                    pypi_0    pypi
google-crc32c             1.1.2                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
google-resumable-media    1.2.0                    pypi_0    pypi
googleapis-common-protos  1.52.0           py37h89c1867_1    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
grpc-cpp                  1.34.1               h146f9af_1    conda-forge
grpc-google-iam-v1        0.12.3                   pypi_0    pypi
grpcio                    1.32.0                   pypi_0    pypi
grpcio-gcp                0.2.2            py37h89c1867_4    conda-forge
gst-plugins-base          1.14.5               h0935bb2_2    conda-forge
gstreamer                 1.18.3               h3560a44_0    conda-forge
h5py                      2.10.0                   pypi_0    pypi
harfbuzz                  2.7.4                h5cf4720_0    conda-forge
hdf5                      1.10.6          nompi_h6a2412b_1114    conda-forge
htmlmin                   0.1.12                     py_1    conda-forge
httplib2                  0.18.1             pyh9f0ad1d_0    conda-forge
icu                       68.1                 h58526e2_0    conda-forge
idna                      2.10                       py_0  
imagehash                 4.2.0              pyhd8ed1ab_0    conda-forge
importlib-metadata        3.4.0            py37h89c1867_0    conda-forge
importlib_metadata        3.4.0                hd8ed1ab_0    conda-forge
ipykernel                 5.3.4            py37h888b3d9_1    conda-forge
ipython                   7.18.1           py37hc6149b9_1    conda-forge
ipython-sql               0.3.9           py37hc8dfbb8_1002    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.6.3              pyhd3deb0d_0    conda-forge
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jedi                      0.18.0           py37h89c1867_2    conda-forge
jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge
jinja2-time               0.2.0                      py_2    conda-forge
joblib                    1.0.0              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonschema                3.2.0                      py_2    conda-forge
jupyter                   1.0.0                    pypi_0    pypi
jupyter-aihub-deploy-extension 0.2                      pypi_0    pypi
jupyter-console           6.2.0                    pypi_0    pypi
jupyter-http-over-ws      0.0.8                    pypi_0    pypi
jupyter-server-proxy      1.5.3                    pypi_0    pypi
jupyter_client            6.1.11             pyhd8ed1ab_1    conda-forge
jupyter_core              4.7.0            py37h89c1867_1    conda-forge
jupyterlab                1.2.16                     py_0    conda-forge
jupyterlab-git            0.11.0                     py_0    conda-forge
jupyterlab-nvdashboard    0.4.0                    pypi_0    pypi
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         1.2.0                      py_0    conda-forge
jupyterlab_widgets        1.0.0              pyhd8ed1ab_1    conda-forge
keras                     2.4.0                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
keras-tuner               1.0.2                    pypi_0    pypi
kiwisolver                1.3.1            py37h2527ec5_1    conda-forge
krb5                      1.17.2               h926e7f8_0    conda-forge
lame                      3.100             h7f98852_1001    conda-forge
lcms2                     2.11                 hcbb858e_1    conda-forge
ld_impl_linux-64          2.35.1               hea4e1c9_1    conda-forge
libblas                   3.9.0                7_openblas    conda-forge
libcblas                  3.9.0                7_openblas    conda-forge
libclang                  11.0.1          default_ha53f305_1    conda-forge
libcurl                   7.71.1               hcdd3856_8    conda-forge
libedit                   3.1.20191231         h14c3975_1  
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.3.0               h2828fa1_18    conda-forge
libgfortran-ng            9.3.0               hff62375_18    conda-forge
libgfortran5              9.3.0               hff62375_18    conda-forge
libglib                   2.66.4               h164308a_1    conda-forge
libgomp                   9.3.0               h2828fa1_18    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0                7_openblas    conda-forge
liblapacke                3.9.0                7_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libllvm11                 11.0.1               hf817b99_0    conda-forge
libnghttp2                1.41.0               h8cfc5f6_2    conda-forge
libopenblas               0.3.12          pthreads_h4812303_1    conda-forge
libopencv                 4.5.0            py37h90094e2_7    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libpq                     12.3                 h255efa7_3    conda-forge
libprotobuf               3.14.0               h780b84a_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libssh2                   1.9.0                hab1572f_5    conda-forge
libstdcxx-ng              9.3.0               h6de172a_18    conda-forge
libthrift                 0.13.0               h5aa387f_6    conda-forge
libtiff                   4.2.0                hdc55705_0    conda-forge
libutf8proc               2.6.1                h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libuv                     1.40.0               h7f98852_0    conda-forge
libwebp-base              1.1.0                h36c2ea0_3    conda-forge
libxcb                    1.13              h7f98852_1003    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.10               h72842e0_3    conda-forge
llvmlite                  0.35.0           py37h9d7f4d0_1    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
markdown                  3.3.3              pyh9f0ad1d_0    conda-forge
markupsafe                1.1.1            py37h5e8e339_3    conda-forge
matplotlib-base           3.3.3            py37h0c9df89_0    conda-forge
missingno                 0.4.2                      py_1    conda-forge
mistune                   0.8.4           py37h5e8e339_1003    conda-forge
multidict                 5.1.0            py37h5e8e339_1    conda-forge
multimethod               1.4                        py_0    conda-forge
mypy_extensions           0.4.3            py37h89c1867_3    conda-forge
mysql-common              8.0.22               ha770c72_3    conda-forge
mysql-libs                8.0.22               h935591d_3    conda-forge
nb_conda                  2.2.1            py37hc8dfbb8_4    conda-forge
nb_conda_kernels          2.3.1            py37h89c1867_0    conda-forge
nbclient                  0.5.1                      py_0    conda-forge
nbconvert                 6.0.7            py37h89c1867_3    conda-forge
nbdime                    2.1.0                      py_0    conda-forge
nbformat                  5.1.2              pyhd8ed1ab_1    conda-forge
ncurses                   6.2                  he6710b0_1  
nest-asyncio              1.4.3              pyhd8ed1ab_0    conda-forge
nettle                    3.6                  he412f7d_0    conda-forge
networkx                  2.5                        py_0    conda-forge
nodejs                    15.3.0               h25f6087_0    conda-forge
notebook                  6.2.0            py37h89c1867_0    conda-forge
notebook-executor         0.2                      pypi_0    pypi
nspr                      4.29                 h9c3ff4c_1    conda-forge
nss                       3.60                 hb5efdd6_0    conda-forge
numba                     0.52.0           py37hdc94413_0    conda-forge
numpy                     1.19.5           py37haa41c4c_1    conda-forge
oauth2client              4.1.3                    pypi_0    pypi
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46               pyh9f0ad1d_1    conda-forge
opencv                    4.5.0            py37h89c1867_7    conda-forge
openh264                  2.1.1                h780b84a_0    conda-forge
openssl                   1.1.1i               h7f98852_0    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
orc                       1.6.6                h7950760_1    conda-forge
packaging                 20.8               pyhd3deb0d_0    conda-forge
pandas                    1.2.1            py37hdc94413_0    conda-forge
pandas-profiling          2.8.0                      py_0    conda-forge
pandoc                    2.11.3.2             h7f98852_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
papermill                 2.3.0              pyhd8ed1ab_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.8.1              pyhd8ed1ab_0    conda-forge
pathspec                  0.8.1              pyhd3deb0d_0    conda-forge
patsy                     0.5.1                      py_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
phik                      0.10.0                     py_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.1.0            py37he6b4880_1    conda-forge
pip                       20.3.3             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
poyo                      0.5.0                      py_0    conda-forge
prettytable               2.0.0              pyhd8ed1ab_0    conda-forge
prometheus_client         0.9.0              pyhd3deb0d_0    conda-forge
promise                   2.3                      pypi_0    pypi
prompt-toolkit            3.0.11             pyha770c72_0    conda-forge
protobuf                  3.14.0           py37hcd2ae1e_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
py-opencv                 4.5.0            py37h888b3d9_7    conda-forge
pyarrow                   2.0.0           py37h9425694_17_cpu    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.8                    pypi_0    pypi
pycosat                   0.6.3           py37h5e8e339_1006    conda-forge
pycparser                 2.20                       py_2  
pyerfa                    1.7.1.1          py37h5e8e339_2    conda-forge
pygments                  2.7.4              pyhd8ed1ab_0    conda-forge
pyjwt                     2.0.1              pyhd8ed1ab_0    conda-forge
pynvml                    8.0.4                    pypi_0    pypi
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyrsistent                0.17.3           py37h5e8e339_2    conda-forge
pysocks                   1.7.1            py37h89c1867_3    conda-forge
python                    3.7.9           hffdb5ce_0_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python-kubernetes         12.0.1             pyhd3deb0d_0    conda-forge
python-slugify            4.0.1              pyh9f0ad1d_0    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
pytz                      2020.5             pyhd8ed1ab_0    conda-forge
pywavelets                1.1.1            py37h902c9e0_3    conda-forge
pyyaml                    5.4.1            py37h5e8e339_0    conda-forge
pyzmq                     21.0.1           py37h499b945_0    conda-forge
qt                        5.12.9               h9d6b050_2    conda-forge
qtconsole                 5.0.1                    pypi_0    pypi
re2                       2020.11.01           h58526e2_0    conda-forge
readline                  8.0                  h7b6447c_0  
regex                     2020.11.13       py37h5e8e339_1    conda-forge
requests                  2.25.1             pyhd3deb0d_0    conda-forge
requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge
retrying                  1.3.3                    pypi_0    pypi
rsa                       4.7                pyhd3deb0d_0    conda-forge
ruamel_yaml               0.15.80         py37h5e8e339_1003    conda-forge
s2n                       0.10.24              h9b69904_0    conda-forge
scikit-learn              0.24.1           py37h69acf81_0    conda-forge
scipy                     1.6.0            py37h14a347d_0    conda-forge
seaborn                   0.11.1               ha770c72_0    conda-forge
seaborn-base              0.11.1             pyhd8ed1ab_0    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                51.3.3                   pypi_0    pypi
simpervisor               0.4                      pypi_0    pypi
simplejson                3.17.2           py37h5e8e339_2    conda-forge
six                       1.15.0             pyh9f0ad1d_0    conda-forge
smmap                     3.0.4              pyh9f0ad1d_0    conda-forge
snappy                    1.1.8                he1b5a44_3    conda-forge
sqlalchemy                1.3.22           py37h5e8e339_1    conda-forge
sqlite                    3.34.0               h74cdb3f_0    conda-forge
sqlparse                  0.4.1              pyh9f0ad1d_0    conda-forge
statsmodels               0.12.1           py37h902c9e0_2    conda-forge
tabulate                  0.8.7                    pypi_0    pypi
tangled-up-in-unicode     0.0.6              pyh9f0ad1d_0    conda-forge
tenacity                  6.3.1              pyhd8ed1ab_0    conda-forge
tensorboard               2.4.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tensorflow                2.4.0                    pypi_0    pypi
tensorflow-cloud          0.1.7                    pypi_0    pypi
tensorflow-datasets       3.0.0                    pypi_0    pypi
tensorflow-estimator      2.4.0                    pypi_0    pypi
tensorflow-hub            0.9.0                    pypi_0    pypi
tensorflow-io             0.17.0                   pypi_0    pypi
tensorflow-metadata       0.26.0                   pypi_0    pypi
tensorflow-probability    0.11.0                   pypi_0    pypi
tensorflow-serving-api    2.4.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
terminado                 0.9.2            py37h89c1867_0    conda-forge
terminaltables            3.1.0                    pypi_0    pypi
testpath                  0.4.4                      py_0    conda-forge
text-unidecode            1.3                        py_0    conda-forge
textwrap3                 0.9.2                      py_0    conda-forge
threadpoolctl             2.1.0              pyh5ca1d4c_0    conda-forge
tk                        8.6.10               hbc83047_0  
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tornado                   6.1              py37h5e8e339_1    conda-forge
tqdm                      4.56.0             pyhd8ed1ab_0    conda-forge
traitlets                 5.0.5                      py_0    conda-forge
typed-ast                 1.4.2            py37h5e8e339_0    conda-forge
typing-extensions         3.7.4.3                       0    conda-forge
typing_extensions         3.7.4.3                    py_0    conda-forge
unidecode                 1.1.2              pyhd8ed1ab_0    conda-forge
uritemplate               3.0.1                      py_0    conda-forge
urllib3                   1.26.2             pyhd8ed1ab_0    conda-forge
visions                   0.7.0              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          0.57.0           py37h89c1867_4    conda-forge
werkzeug                  1.0.1                    pypi_0    pypi
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
whichcraft                0.6.1                      py_0    conda-forge
widgetsnbextension        3.5.1            py37h89c1867_4    conda-forge
witwidget                 1.7.0                    pypi_0    pypi
wrapt                     1.12.1           py37h5e8e339_3    conda-forge
x264                      1!161.3030           h7f98852_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.12               h516909a_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h7b6447c_0  
yaml                      0.2.5                h7b6447c_0  
yarl                      1.6.3            py37h5e8e339_1    conda-forge
zeromq                    4.3.3                h58526e2_3    conda-forge
zipp                      3.4.0                      py_0    conda-forge
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.8                ha95c52a_1    conda-forge

[jacobtomlinson]

Many thanks @gogasca.

Its appears that the websocket connection is successful, but Bokeh isn't happy about something.

bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'

I expect some critical header is being rewritten somewhere during the multiple layers of proxies.

It appears this issue came up once in the jupyter-server-proxy and was fixed by correctly proxying subprotocols. I wonder if those subprotocols are not being proxied by Google Cloud Inverse Proxy.

Is this something you are able to investigate?

[gogasca]

Yes, I should be able to run a local proxy version and test. Proxy do not support websockets but it does have a shim to enable support. https://github.com/google/inverting-proxy/blob/a121408eb881af9e75b10064b46258c4ab12c47f/agent/websockets/shim.go#L41

I will test and get back to you. Thanks

[eriknil-onehot]

@gogasca did you have any luck with this? I'm experiencing the same issue myself.