Open consideRatio opened 9 months ago
I think I'm observing this with 4.1.1 and jupyterlab-nvdashboard, which uses bokeh:
[W 2024-03-13 06:19:15.107 SingleUserLabApp handlers:539] Websocket subprotocol between proxy/server (None)
became different than for client/proxy (bokeh) due to https://github.com/jupyterhub/jupyter-server-proxy/issues/459.
Requested subprotocols were ['bokeh'].
This causes bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'
And it worked before in 4.1.0?
Yes it was working before w/4.1.0
@rcthomas can you help me reproduce this?
So far I've not seen the warning in jupyter server logs, or in javascript console logs, and the dashboard works for me. I've tried this locally and with a locally run jupyterhub so far.
Where was the bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'
error printed out - jupyter_server or browser javascript console logs?
Thanks @consideRatio! Relieved to see there's a configuration out there that's known to work.
I observed the message in the Jupyter server logs. I've turned on debug-level logging. Here's the log output when I try to open the machine resources tab. Is the "client sent protocols" line what you would expect?
The warnings about "legend" apparently come from us not yet upgrading the nvdashboard to 0.9, but for now I am leaving 0.8 in place. Looking at the diff there I don't think that would fix what I'm seeing.
I can revert and add the log details for that if it helps. Here is a summary of the environment in case that helps trace down any differences in our environments:
Thank you @rcthomas! I could reproduce this now and conclude that the surgical change between 4.1.1 and 4.1.0 makes this difference as well.
I'll work this with highest prio since we need a release out with things in 4.1.1, but if it doesn't work thats trouble.
Okay I've figured out the issue, but I'm out of time to work this further today, will continue tomorrow or if I find free time.
Wow thanks! But if you find free time, use it for yourself!
@consideRatio #462 appears to have fixed it! Thanks!
Thank you for the quick testing and reporting @rcthomas!
Looking at #442 led me to conclude that when a websocket connection is being proxied, we establish the client/proxy socket fully before establishing a proxy/server connection. This is a problem, because when a websocket is established, a client requests subprotocols it knows how to speak, and the server is to choose one it supports. The choice of subprotocol influences the format of the sent messages, so its critical that the server we proxy to gets to decide this. Currently though, we just pick the first subprotocol instead and hope that the server we proxy to will later accept it.
In #458 I added test cases to confirm our bugged behavior, and test cases we could switch to if we resolve this by not finalizing the client/proxy handshake before we establish a server/proxy handshake.
This bug is probably not very problematic, and with #458, I'm looking to provide a warning in case it could be a problem. The problem would arise mostly if the client doesn't know how to correctly interact with the server, so I think this can be left unresolved until there is a clear example of when this becomes relevant. The complexity to solve this bug could be notable.
Related