jupyterhub / jupyter-server-proxy

Jupyter notebook server extension to proxy web services.
https://jupyter-server-proxy.readthedocs.io
BSD 3-Clause "New" or "Revised" License
344 stars 147 forks source link

WebSocket subprotocols for client/proxy are chosen without asking the server we proxy to #459

Open consideRatio opened 6 months ago

consideRatio commented 6 months ago

Looking at #442 led me to conclude that when a websocket connection is being proxied, we establish the client/proxy socket fully before establishing a proxy/server connection. This is a problem, because when a websocket is established, a client requests subprotocols it knows how to speak, and the server is to choose one it supports. The choice of subprotocol influences the format of the sent messages, so its critical that the server we proxy to gets to decide this. Currently though, we just pick the first subprotocol instead and hope that the server we proxy to will later accept it.

In #458 I added test cases to confirm our bugged behavior, and test cases we could switch to if we resolve this by not finalizing the client/proxy handshake before we establish a server/proxy handshake.

This bug is probably not very problematic, and with #458, I'm looking to provide a warning in case it could be a problem. The problem would arise mostly if the client doesn't know how to correctly interact with the server, so I think this can be left unresolved until there is a clear example of when this becomes relevant. The complexity to solve this bug could be notable.

Related

rcthomas commented 5 months ago

I think I'm observing this with 4.1.1 and jupyterlab-nvdashboard, which uses bokeh:

[W 2024-03-13 06:19:15.107 SingleUserLabApp handlers:539] Websocket subprotocol between proxy/server (None)
became different than for client/proxy (bokeh) due to https://github.com/jupyterhub/jupyter-server-proxy/issues/459.
Requested subprotocols were ['bokeh'].

This causes bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'

consideRatio commented 5 months ago

And it worked before in 4.1.0?

rcthomas commented 5 months ago

Yes it was working before w/4.1.0

consideRatio commented 5 months ago

@rcthomas can you help me reproduce this?

So far I've not seen the warning in jupyter server logs, or in javascript console logs, and the dashboard works for me. I've tried this locally and with a locally run jupyterhub so far.

Where was the bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh' error printed out - jupyter_server or browser javascript console logs?

rcthomas commented 5 months ago

Thanks @consideRatio! Relieved to see there's a configuration out there that's known to work.

I observed the message in the Jupyter server logs. I've turned on debug-level logging. Here's the log output when I try to open the machine resources tab. Is the "client sent protocols" line what you would expect?

log snippet ``` [D 2024-03-13 08:15:03.842 SingleUserLabApp handlers:381] Proxying request to http://localhost:57475/Machine-Resources [D 2024-03-13 08:15:03.844 SingleUserLabApp handlers:381] Proxying request to http://localhost:57475/Machine-Resources BokehDeprecationWarning: 'legend' keyword is deprecated, use explicit 'legend_label', 'legend_field', or 'legend_group' keywords instead BokehDeprecationWarning: 'legend' keyword is deprecated, use explicit 'legend_label', 'legend_field', or 'legend_group' keywords instead BokehDeprecationWarning: 'legend' keyword is deprecated, use explicit 'legend_label', 'legend_field', or 'legend_group' keywords instead BokehDeprecationWarning: 'legend' keyword is deprecated, use explicit 'legend_label', 'legend_field', or 'legend_group' keywords instead BokehDeprecationWarning: 'legend' keyword is deprecated, use explicit 'legend_label', 'legend_field', or 'legend_group' keywords instead BokehDeprecationWarning: 'legend' keyword is deprecated, use explicit 'legend_label', 'legend_field', or 'legend_group' keywords instead BokehDeprecationWarning: 'legend' keyword is deprecated, use explicit 'legend_label', 'legend_field', or 'legend_group' keywords instead BokehDeprecationWarning: 'legend' keyword is deprecated, use explicit 'legend_label', 'legend_field', or 'legend_group' keywords instead [D 2024-03-13 08:15:04.275 SingleUserLabApp handlers:592] Client sent subprotocols: ['bokeh', 'eyJzZXNzaW9uX2lkIjogIjFsQU1XNHQ2SUtNdjlqU1BBNG1QZHpIdktVOUN1VXlhSEFXOE5COUFtTXVvIiwgInNlc3Npb25fZXhwaXJ5IjogMTcxMDM0MzIwMywgIl9fYmtfX3psaWJfIjogImVOcWxWV2x2MnpnUV9TdUNQcVZkVXlZcFVZZURZcEY0YzhkcE5zN2hCQUVDa2hvZEcwbFVKVGwxMHVTX2w1S2N4azdSWFN3S3dZQThiOTdNNF9DTl9jMU1nSWRRMWViSS1HYnVxN3JSTC1ZXzhfS3hnY29xTkNDdFdEMllBOE84cUtGQ1d6RVVYY3BFUGFWWnhvZk13c2JHaE11MGFGU2RiQm9IUlFPWm9RUEc1Nmt4TXdpLUktek8tMkJzbFdVR1Z5Q08wbWJJYk0teVhXUGphUDk4Y2p3d3N2UWVqRDJROS1xRE1VNHFsY09RVUdyaDlqR21QT0pWdXFTME1yYWtoTEtUME1DaUdTWk5uZzI0THA1SzNxU3FHQzdheUItTDk5RTgyX3p5Q1Z2QklNMTVERVAta0ViTDE2OGd5dGRvV2NTRGo4T1BYYXFfVnFCTzR3SkNCQXVaOENLR3pZZFB3dTdTdkRkUmFLZVFLa3lMdUZVWFA2WGx3QWdoeW5nREEwTlVBLU9wYnNLVjdHTmRhSzdidHRsUW9JdnBBSXBlWkpzMEJZbkdDYnJnTFh4cmRvTko1X210cVZ2Zm1ucEF0enJyMWp4UnpjYldhTHZpUmJpRXFOTWplMHJGR1N3bnVrcGJxNDRtU3FSWnAtRlB2STZjYXVXUnF2SmVRTTdsNS1rYmVSY2FtYUNfb0RkTUdsVThoM1Zzb3NLdWJLRm5IZXNacktQVHRPblFXdk9RcXRJNExkWVRXcnQxcWtobnZqS3V0RV9SUVZHRG5GZUF6dURMWERkdmZXdDJHVE8wcTZxdnZBcjFMZW0zTGs1OXk3WUlkaTJLM1lIMm91VlFLN0FJd2U4Sl8ycjgxY1JUVlhXSmptUF9CRldxVVMyV05FMVp2MGVuTW9FY2ZnMUQ5ZENmdDZrNFJPazljcG1VUWVDSHZrUmhGQ1JlenpnRG5xR0Q4cWZUdmFMZFVOQkIyQ2FFUVJDRkFYVmtKTUxJOFNQR2NCUUpHbUxYNVpod3YtZjhoN0N4S3RvOWFfSGhYRl9Kc0dvU2xmTjZXRUtWNWZOR1R3dGxTdDhlS3ZSOUk4RnJHQllQSWE4VG9YU0Z2cHdlemVKUjlwVkszaVNfVjIxMTVIcFVpOS1wcG85WGdHdzN2SzBpTTFXRC1hTGpVcW43RlBwZnhidEZYVVV0VEpfRFFESmhSX2laZWRnUFNjUnBCQjRSbVBuQ0UxUlExMmZjczIwM2VDWWV3U1NnTnVuTXVmUlZNaGVvRllxV1F0R3ZoZmI5eUFnX0V6eHFpOWtPOVgzZEdJXy1mN0ZuQjQ4bTlKcWU3TnpNVDU3Q1hZa1ByNmJuWjQ4bjUwbDVsVl9PcHRraEYzbHlla0p2dHA4OURDSUl3YUdSSU9BRVhIc0ZpTkJMSTJ4RzdTQml2czBpajlpQjV6RWlBS2puZzVENnNDUmlybU9MOXJ4RlhqZThNdzBqVVNnWWR4QjJwQlpEc1VUVWlTUnl0Ujk5a01JUmZzZTRpX25kOFM2LVBCcTczdloxeTl5YkVrc19ucTZzUDlpeDJyLUVIMS1EX3ZfaGxYbzRQZzNPcnFjblI3dHIxTUIzN1paS1BQb2o0RENiclpMVGJsbjJ0b2h1b0ROYzdQb08wWXZWWFNDbVR2RGFnOTdzMnNGc3RyUF85Mm9QZ2lsaHhHMmxzWldBNzZ3SmZHMUJMSTk1anVPelFCTmNYeTg0OWZ2dGJUdk1ibmJPY1RDZTdGLS02eEQ0bU9rOXAzUWw0TnR2SFZZc1VVTmRhenVqX2xqTTB4NlZvZXVCVHh5QmdSUFhkckhVNWcyb3ZydklmSG41RHU2X1pSYyJ9'], selecting the first [I 2024-03-13 08:15:04.276 SingleUserLabApp handlers:520] Trying to establish websocket connection to ws://localhost:57475/Machine-Resources/ws [I 2024-03-13 08:15:04.279 SingleUserLabApp handlers:534] Websocket connection established to ws://localhost:57475/Machine-Resources/ws [W 2024-03-13 08:15:04.279 SingleUserLabApp handlers:539] Websocket subprotocol between proxy/server (None) became different than for client/proxy (bokeh) due to https://github.com/jupyterhub/jupyter-server-proxy/issues/459. Requested subprotocols were ['bokeh']. Uncaught exception GET /Machine-Resources/ws (::1) HTTPServerRequest(protocol='http', host='jupyter.nersc.gov', method='GET', uri='/Machine-Resources/ws', version='HTTP/1.1', remote_ip='::1') Traceback (most recent call last): File "/global/common/software/nersc/current/jupyter/ex/23-10/lib/python3.10/site-packages/tornado/websocket.py", line 937, in _accept_connection open_result = handler.open(*handler.open_args, **handler.open_kwargs) File "/global/common/software/nersc/current/jupyter/ex/23-10/lib/python3.10/site-packages/tornado/web.py", line 3290, in wrapper return method(self, *args, **kwargs) File "/global/common/software/nersc/current/jupyter/ex/23-10/lib/python3.10/site-packages/bokeh/server/views/ws.py", line 137, in open raise ProtocolError("Subprotocol header is not 'bokeh'") bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh' ```

The warnings about "legend" apparently come from us not yet upgrading the nvdashboard to 0.9, but for now I am leaving 0.8 in place. Looking at the diff there I don't think that would fix what I'm seeing.

I can revert and add the log details for that if it helps. Here is a summary of the environment in case that helps trace down any differences in our environments:

environment ``` # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge additional-resources-menu 1.0.0 pypi_0 pypi aiofiles 22.1.0 pyhd8ed1ab_0 conda-forge aiohttp 3.8.6 py310h2372a71_1 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge aiosqlite 0.19.0 pyhd8ed1ab_0 conda-forge alembic 1.12.1 pyhd8ed1ab_0 conda-forge anyio 4.0.0 pyhd8ed1ab_0 conda-forge aom 3.6.1 h59595ed_0 conda-forge archspec 0.2.1 pyhd8ed1ab_1 conda-forge argon2-cffi 23.1.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py310h2372a71_4 conda-forge arrow 1.3.0 pyhd8ed1ab_0 conda-forge asttokens 2.4.1 pyhd8ed1ab_0 conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge async_generator 1.10 py_0 conda-forge attrs 23.1.0 pyh71513ae_1 conda-forge autopep8 2.0.4 pyhd8ed1ab_0 conda-forge babel 2.13.1 pyhd8ed1ab_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge batchspawner 1.2.0 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.12.2 pyha770c72_0 conda-forge black 23.10.1 py310hff52083_0 conda-forge bleach 6.1.0 pyhd8ed1ab_0 conda-forge blinker 1.6.3 pyhd8ed1ab_0 conda-forge blosc 1.21.5 h0f2a231_0 conda-forge bokeh 2.4.3 pyhd8ed1ab_3 conda-forge boltons 23.0.0 pyhd8ed1ab_0 conda-forge brotli 1.1.0 hd590300_0 conda-forge brotli-bin 1.1.0 hd590300_0 conda-forge brotli-python 1.1.0 py310hc6cd4ac_0 conda-forge brunsli 0.1 h9c3ff4c_0 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.19.1 hd590300_0 conda-forge c-blosc2 2.10.5 hb4ffafa_0 conda-forge ca-certificates 2024.2.2 hbcca054_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge certifi 2024.2.2 pyhd8ed1ab_0 conda-forge certipy 0.1.3 py_0 conda-forge cffi 1.15.1 py310h255011f_3 conda-forge charls 2.4.2 h59595ed_0 conda-forge charset-normalizer 3.2.0 pyhd8ed1ab_0 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge cloudpickle 3.0.0 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge comm 0.1.4 pyhd8ed1ab_0 conda-forge conda 23.9.0 py310hff52083_2 conda-forge conda-libmamba-solver 23.9.3 pyhd8ed1ab_0 conda-forge conda-package-handling 2.2.0 pyh38be061_0 conda-forge conda-package-streaming 0.9.0 pyhd8ed1ab_0 conda-forge configurable-http-proxy 4.5.6 h92b4e83_1 conda-forge contourpy 1.1.1 py310hd41b1e2_1 conda-forge cryptography 41.0.3 py310h75e40e8_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge cytoolz 0.12.2 py310h2372a71_1 conda-forge dask-core 2023.10.1 pyhd8ed1ab_0 conda-forge dask-labextension 6.2.0 pyhd8ed1ab_0 conda-forge dav1d 1.2.1 hd590300_0 conda-forge debugpy 1.8.0 py310hc6cd4ac_1 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge deprecation 2.1.0 pypi_0 pypi distributed 2023.10.1 pyhd8ed1ab_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge exceptiongroup 1.1.3 pyhd8ed1ab_0 conda-forge executing 2.0.1 pyhd8ed1ab_0 conda-forge fmt 10.1.1 h00ab1b0_0 conda-forge fonttools 4.43.1 py310h2372a71_0 conda-forge fqdn 1.5.1 pyhd8ed1ab_0 conda-forge freetype 2.12.1 h267a509_2 conda-forge frozenlist 1.4.0 py310h2372a71_1 conda-forge fsspec 2023.10.0 pyhca7485f_0 conda-forge giflib 5.2.1 h0b41bf4_3 conda-forge gitdb 4.0.11 pyhd8ed1ab_0 conda-forge gitpython 3.1.40 pyhd8ed1ab_0 conda-forge gmp 6.2.1 h58526e2_0 conda-forge greenlet 3.0.1 py310hc6cd4ac_0 conda-forge h5grove 1.3.0 pyhd8ed1ab_0 conda-forge h5py 3.10.0 nompi_py310ha2ad45a_100 conda-forge hdf5 1.14.2 nompi_h4f84152_100 conda-forge icu 73.2 h59595ed_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge imagecodecs 2023.9.18 py310h3a85d3a_0 conda-forge importlib-metadata 6.8.0 pyha770c72_0 conda-forge importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge importlib_resources 6.1.0 pyhd8ed1ab_0 conda-forge ipykernel 6.26.0 pyhf8b6a83_0 conda-forge ipympl 0.9.3 pyhd8ed1ab_0 conda-forge ipython 8.16.1 pyh0d859eb_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 8.1.1 pyhd8ed1ab_0 conda-forge isoduration 20.11.0 pyhd8ed1ab_0 conda-forge isort 5.12.0 pyhd8ed1ab_1 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge json5 0.9.14 pyhd8ed1ab_0 conda-forge jsonpatch 1.32 pyhd8ed1ab_0 conda-forge jsonpointer 2.0 py_0 conda-forge jsonschema 4.19.2 pyhd8ed1ab_0 conda-forge jsonschema-specifications 2023.7.1 pyhd8ed1ab_0 conda-forge jsonschema-with-format-nongpl 4.19.2 pyhd8ed1ab_0 conda-forge jupyter-kernel-cloner 0.0.4 pypi_0 pypi jupyter-packaging 0.12.3 pypi_0 pypi jupyter-resource-usage 0.7.1 pyhd8ed1ab_0 conda-forge jupyter-server-mathjax 0.2.6 pyh5bfe37b_1 conda-forge jupyter-server-proxy 4.1.1 pyhd8ed1ab_0 conda-forge jupyter_client 7.4.9 pyhd8ed1ab_0 conda-forge jupyter_core 5.5.0 py310hff52083_0 conda-forge jupyter_events 0.8.0 pyhd8ed1ab_0 conda-forge jupyter_server 2.9.1 pyhd8ed1ab_0 conda-forge jupyter_server_fileid 0.9.0 pyhd8ed1ab_0 conda-forge jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge jupyter_server_ydoc 0.8.0 pyhd8ed1ab_0 conda-forge jupyter_telemetry 0.1.0 pyhd8ed1ab_1 conda-forge jupyter_ydoc 0.2.4 pyhd8ed1ab_0 conda-forge jupyterhub 2.3.1 pyhd8ed1ab_0 conda-forge jupyterhub-base 2.3.1 pyhd8ed1ab_0 conda-forge jupyterlab 3.6.6 pyhd8ed1ab_0 conda-forge jupyterlab-favorites 3.1.1 pyhd8ed1ab_0 conda-forge jupyterlab-git 0.44.0 pyhd8ed1ab_0 conda-forge jupyterlab-h5web 8.0.0 pyhd8ed1ab_1 conda-forge jupyterlab-nvdashboard 0.8.0 pyhc268e32_0 conda-forge jupyterlab-recents 3.2.0 pyhd8ed1ab_0 conda-forge jupyterlab-slurm 3.0.1 pypi_0 pypi jupyterlab-spreadsheet-editor 0.6.1 pyhd8ed1ab_0 conda-forge jupyterlab_code_formatter 2.2.1 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge jupyterlab_server 2.25.0 pyhd8ed1ab_0 conda-forge jupyterlab_vim 0.16.0 pyhd8ed1ab_0 conda-forge jupyterlab_widgets 3.0.9 pyhd8ed1ab_0 conda-forge jupytext 1.15.2 pyh5da7574_0 conda-forge jxrlib 1.1 h7f98852_2 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.5 py310hd41b1e2_1 conda-forge krb5 1.21.2 h659d440_0 conda-forge lcms2 2.15 h7f713cb_2 conda-forge ld_impl_linux-64 2.40 h41732ed_0 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libaec 1.1.2 h59595ed_1 conda-forge libarchive 3.7.2 h039dbb9_0 conda-forge libavif16 1.0.1 h87da1f6_2 conda-forge libblas 3.9.0 19_linux64_openblas conda-forge libbrotlicommon 1.1.0 hd590300_0 conda-forge libbrotlidec 1.1.0 hd590300_0 conda-forge libbrotlienc 1.1.0 hd590300_0 conda-forge libcblas 3.9.0 19_linux64_openblas conda-forge libcurl 8.4.0 hca28451_0 conda-forge libdeflate 1.19 hd590300_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 13.1.0 he5830b7_0 conda-forge libgfortran-ng 13.2.0 h69a702a_0 conda-forge libgfortran5 13.2.0 ha4646dd_0 conda-forge libgomp 13.1.0 he5830b7_0 conda-forge libiconv 1.17 h166bdaf_0 conda-forge libjpeg-turbo 2.1.5.1 hd590300_1 conda-forge liblapack 3.9.0 19_linux64_openblas conda-forge libmamba 1.5.2 had39da4_0 conda-forge libmambapy 1.5.2 py310h39ff949_0 conda-forge libnghttp2 1.52.0 h61bc06f_0 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libopenblas 0.3.24 pthreads_h413a1c8_0 conda-forge libpng 1.6.39 h753d276_0 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libsolv 0.7.24 hfc55251_3 conda-forge libsqlite 3.43.0 h2797004_0 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-ng 13.1.0 hfd8a6a1_0 conda-forge libtiff 4.6.0 h29866fb_1 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libuv 1.46.0 hd590300_0 conda-forge libwebp-base 1.3.2 hd590300_0 conda-forge libxcb 1.15 h0b41bf4_0 conda-forge libxml2 2.11.5 h232c23b_1 conda-forge libzlib 1.2.13 hd590300_5 conda-forge libzopfli 1.0.3 h9c3ff4c_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge lzo 2.10 h516909a_1000 conda-forge mako 1.2.4 pyhd8ed1ab_0 conda-forge mamba 1.5.2 py310h51d5547_0 conda-forge markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py310h2372a71_1 conda-forge matplotlib-base 3.8.0 py310h62c0568_1 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mdit-py-plugins 0.4.0 pyhd8ed1ab_0 conda-forge mdurl 0.1.0 pyhd8ed1ab_0 conda-forge mistune 3.0.1 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.6 py310hd41b1e2_0 conda-forge multidict 6.0.4 py310h2372a71_1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mypy_extensions 1.0.0 pyha770c72_0 conda-forge nbclassic 1.0.0 pyhb4ecaf3_1 conda-forge nbclient 0.8.0 pyhd8ed1ab_0 conda-forge nbconvert 7.10.0 pyhd8ed1ab_0 conda-forge nbconvert-core 7.10.0 pyhd8ed1ab_0 conda-forge nbconvert-pandoc 7.10.0 pyhd8ed1ab_0 conda-forge nbdime 3.2.1 pyhd8ed1ab_0 conda-forge nbformat 5.9.2 pyhd8ed1ab_0 conda-forge ncurses 6.4 hcb278e6_0 conda-forge nersc-refresh-announcements 0.1.0 pypi_0 pypi nest-asyncio 1.5.8 pyhd8ed1ab_0 conda-forge nodejs 20.8.1 h1990674_0 conda-forge notebook 6.5.6 pyha770c72_0 conda-forge notebook-shim 0.2.3 pyhd8ed1ab_0 conda-forge numpy 1.26.0 py310hb13e2d6_0 conda-forge oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge openjpeg 2.5.0 h488ebb8_3 conda-forge openssl 3.2.1 hd590300_0 conda-forge orjson 3.9.10 py310h1e2579a_0 conda-forge overrides 7.4.0 pyhd8ed1ab_0 conda-forge packaging 23.1 pyhd8ed1ab_0 conda-forge pamela 1.1.0 pyh1a96a4e_0 conda-forge pandoc 3.1.3 h32600fe_0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge param 2.0.0 pyhca7485f_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.4.1 pyhd8ed1ab_0 conda-forge pathspec 0.11.2 pyhd8ed1ab_0 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.0.1 py310h29da1c1_1 conda-forge pip 23.2.1 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge platformdirs 3.11.0 pyhd8ed1ab_0 conda-forge plotly 5.18.0 pyhd8ed1ab_0 conda-forge pluggy 1.3.0 pyhd8ed1ab_0 conda-forge prometheus_client 0.18.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.39 pyha770c72_0 conda-forge prompt_toolkit 3.0.39 hd8ed1ab_0 conda-forge psutil 5.9.5 py310h2372a71_1 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pybind11-abi 4 hd8ed1ab_3 conda-forge pycodestyle 2.11.1 pyhd8ed1ab_0 conda-forge pycosat 0.6.4 py310h5764c6d_1 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pycurl 7.45.1 py310h60f9ec7_3 conda-forge pygments 2.16.1 pyhd8ed1ab_0 conda-forge pyjwt 2.8.0 pyhd8ed1ab_0 conda-forge pynvml 11.5.0 pyhd8ed1ab_0 conda-forge pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.10.12 hd12c33a_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.18.1 pyhd8ed1ab_0 conda-forge python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge python_abi 3.10 3_cp310 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pyviz_comms 2.3.2 pyhd8ed1ab_0 conda-forge pyyaml 6.0.1 py310h2372a71_1 conda-forge pyzmq 24.0.1 py310h330234f_1 conda-forge rav1e 0.6.6 he8a937b_2 conda-forge readline 8.2 h8228510_1 conda-forge referencing 0.30.2 pyhd8ed1ab_0 conda-forge reproc 14.2.4 h0b41bf4_0 conda-forge reproc-cpp 14.2.4 hcb278e6_0 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge rpds-py 0.10.6 py310hcb5633a_0 conda-forge ruamel.yaml 0.17.32 py310h2372a71_0 conda-forge ruamel.yaml.clib 0.2.7 py310h1fa729e_1 conda-forge send2trash 1.8.2 pyh41d4057_0 conda-forge setuptools 68.1.2 pyhd8ed1ab_0 conda-forge simpervisor 1.0.0 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge smmap 5.0.0 pyhd8ed1ab_0 conda-forge snappy 1.1.10 h9fff704_0 conda-forge sniffio 1.3.0 pyhd8ed1ab_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.5 pyhd8ed1ab_1 conda-forge sqlalchemy 2.0.22 py310h2372a71_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge svt-av1 1.7.0 h59595ed_0 conda-forge tblib 2.0.0 pyhd8ed1ab_0 conda-forge tenacity 8.2.3 pyhd8ed1ab_0 conda-forge terminado 0.17.1 pyh41d4057_0 conda-forge tifffile 2023.9.26 pyhd8ed1ab_0 conda-forge tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge tomlkit 0.12.1 pypi_0 pypi toolz 0.12.0 pyhd8ed1ab_0 conda-forge tornado 6.3.3 py310h2372a71_1 conda-forge tqdm 4.66.1 pyhd8ed1ab_0 conda-forge traitlets 5.13.0 pyhd8ed1ab_0 conda-forge truststore 0.8.0 pyhd8ed1ab_0 conda-forge types-python-dateutil 2.8.19.14 pyhd8ed1ab_0 conda-forge typing-extensions 4.8.0 hd8ed1ab_0 conda-forge typing_extensions 4.8.0 pyha770c72_0 conda-forge typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge unicodedata2 15.1.0 py310h2372a71_0 conda-forge uri-template 1.3.0 pyhd8ed1ab_0 conda-forge urllib3 2.0.4 pyhd8ed1ab_0 conda-forge wcwidth 0.2.8 pyhd8ed1ab_0 conda-forge webcolors 1.13 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 pyhd8ed1ab_2 conda-forge webio-jupyter-extension 0.1.0 pyhd8ed1ab_0 conda-forge websocket-client 1.6.4 pyhd8ed1ab_0 conda-forge wheel 0.41.2 pyhd8ed1ab_0 conda-forge widgetsnbextension 4.0.9 pyhd8ed1ab_0 conda-forge xorg-libxau 1.0.11 hd590300_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xz 5.2.6 h166bdaf_0 conda-forge y-py 0.5.9 py310h4426083_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge yaml-cpp 0.8.0 h59595ed_0 conda-forge yapf 0.40.1 pyhd8ed1ab_0 conda-forge yarl 1.9.2 py310h2372a71_1 conda-forge ypy-websocket 0.8.2 pyhd8ed1ab_0 conda-forge zeromq 4.3.5 h59595ed_0 conda-forge zfp 1.0.0 h59595ed_4 conda-forge zict 3.0.0 pyhd8ed1ab_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hd590300_5 conda-forge zlib-ng 2.0.7 h0b41bf4_0 conda-forge zstandard 0.19.0 py310h5764c6d_0 conda-forge zstd 1.5.5 hfc55251_0 conda-forge ```
consideRatio commented 5 months ago

Thank you @rcthomas! I could reproduce this now and conclude that the surgical change between 4.1.1 and 4.1.0 makes this difference as well.

I'll work this with highest prio since we need a release out with things in 4.1.1, but if it doesn't work thats trouble.

consideRatio commented 5 months ago

Okay I've figured out the issue, but I'm out of time to work this further today, will continue tomorrow or if I find free time.

rcthomas commented 5 months ago

Wow thanks! But if you find free time, use it for yourself!

rcthomas commented 5 months ago

@consideRatio #462 appears to have fixed it! Thanks!

consideRatio commented 5 months ago

Thank you for the quick testing and reporting @rcthomas!