Open krinsman opened 6 years ago
Here's what seems to be a convincing guess to me right now:
The key/relevant distinction in moving from the SSH port forwarding setup to the JupyterLabHub set up is that the number of remote hosts increases by 1.
Specifically, for the JupyterLabHub setup, there are now two remote hosts, not just one: one remote host running the JupyterHub instance, and another remote host running the JupyterLab instance.
Based on the information written above, as well as experimentation with JupyterHub I did today, it seems most likely to me that the "base route" for the remote host running the JupyterHub instance is "https://hubwebsite/hub/".
My guess then is that the "base route" for the remote host running the JupyterLab instance is "https://hubwebsite/lab/user", but I have less reason to be certain of that.
Anyway, command1
is a valid operation (corresponds to a binary in the path) on the remote host where the JupyterLab instance is located. However, it is not a valid operation on the remote host where the JupyterHub instance is located.
Therefore, my guess is that the problem is the following: the HTTP requests sent via Ajax for whatever reason only connect with/run on the remote host running the JupyterHub instance, while the XMLHttpRequest have the expected behavior of sending the HTTP requests to the remote host running the JupyterLab instance. Therefore, command1
and command2
are being run on two different remote hosts, explaining their different routes. Since they command1
and command2
need to run on the same remote host for things to work, this would seem to explain the observed errors.
Note that this wasn't a problem in the SSH port forwarding case, where there was only one remote host (the same remote host for which command1
is a valid operation), so both Ajax and XMLHttpRequest sent their HTTP requests to it, so the resulting routes looked the same, and command1
was ran on a remote host that could actually execute it.
If this is correct (is it correct based on your experience?), it still seems to leave at least two questions unresolved:
command1route_pattern = create_full_route_pattern_from('/hub/command1')
command2route_pattern = create_full_route_pattern_from('/lab/user/command2')
Or does the notebook server extension see "https://hubwebsite/lab/user" or "https://hubwebsite/hub/" as the "base route"? (In which case the required modifications of the route patterns would be a little trickier to implement.) Or does the notebook server extension see something entirely different (compared to both of the remote hosts) as the "base route", making it much more difficult to choose the correct routing patterns.
I suppose that the answer to 2. doesn't matter as much if 1. remains unanswered, because even if the notebook extension has the correct routes to handle both command1
and command2
, the extension still won't work until both commands are being run on the same remote host (the one where the JupyterLab instance is).
Right now I really don't have any idea how I could change the Ajax code to change where it sends the HTTP requests to. (It's not my choice or preference to use Ajax to send the requests, per se, but it Ajax is what is used by the JavaScript library needed to implement the extension.)
Since the code in question is public now, I can link to it: https://github.com/NERSC/jupyterlab-slurm
Calls to squeue
are being sent via Ajax through the DataTables library, while calls to sbatch
(and also scancel
and scontrol
, but for simplicity I didn't mention them) are being sent via XMLHttpRequest
.
(The ideal would be in the future to update the ones using XMLHttpRequest
to use fetch
, but as has been noted before that was something there hasn't been time for me to figure out yet. Changing the other call to use something besides Ajax seems like it would involve forking the DataTables code and doing major surgery on it which of course would be substantially unpleasant.)
TL;DR: command1
= squeue
and command2
= sbatch
.
That being said, my impression is that what was originally written above is probably simpler to understand than the issue as it unfolds in the actual code. @blink1073
I don't have the bandwidth to maintain this library other than bumping the version along. Happy to accept any PRs.
I have a JupyterLab extension and corresponding notebook server extension which I am trying to use with JupyterLabHub. However, the behavior is substantially different from running JupyterLab on a remote host and using SSH port forwarding. In particular, my extension works in the latter case, but it does not work with JupyterLabHub.
The reason seems to be that the routing mechanisms are confused.
Below I give some details so that the source of my confusion is understandable. My goal was to avoid overloading with unnecessary information, but I apologize in advance if there is still too much.
On the other hand, please also let me know if you require more information from me.
Description of Problem(s): There are two commands, call them
command1
(GET) andcommand2
(POST).When I run the JupyterLab extension on a remote host and then open it in my browser via SSH port forwarding (
ssh -L 8888:0.0.0.0:8888 remote_host
) everything works.With SSH port forwarding on the remote host, according to Google Chrome, the GET requests for command1 have the name
command1
which expands to (according to Chrome when hovering over it in the Network tab of Developer tools) the routehttp://localhost:8888/command1
. Similarly, the POST requests for command2 are have the namecommand2
which expands to the routehttp://localhost:8888/command2
.However, when running the extension with JupyterLabHub on a website (with a different remote host) with JupyterHub correctly installed (it was recently upgraded from vanilla JupyterHub to JupyterLabHub), neither
command1
norcommand2
works. According to Google Chrome:command1
, but the route is now expanded tohttps://hubwebsite/hub/command1
.command2
, but the route is now expanded tohttps://hubwebsite/lab/user/command2
.(Not sure what the correct terminology is here -- my point is simply that, given the behavior on the remote host with SSH port forwarding, I would have expected the routes to have expanded to either:
https://hubwebsite/hub/command1
andhttps://hubwebsite/hub/command2
ORhttps://hubwebsite/lab/user/command1
andhttps://hubwebsite/lab/user/command2
.Instead, it's a mix-and-match for some reason.)
Note: Hypothetically there could be a problem with the authentication/token mechanism I'm using not extending to the second configuration, and the routing is working correctly for command2. However, my understanding is that Tornado describes any 404 errors for POST requests as being 405 errors for security reasons. Also, since the routing isn't working for command1, it seems more likely that this is just a routing problem for command 2, and not actually an authorization problem, as Tornado would have me believe. Yet since I am not sure, I say "Problem(s)" above, not "Problem".
(Additionally still, since JupyterLabHub uses
PageConfig
fromcoreuitls
successfully, it seems unlikely that the same code/module would fail for authentication with JupyberLabHub.)Background/Details: In the Python code for the notebook server extension, there are custom Tornado request handlers (both inheriting from
IPythonHandler
) for each,Command1Handler
andCommand2Handler
.Command1Handler
accepts GET requests, andCommand2Handler
accepts POST requests.Here is (essentially) my
load_jupyter_server_extension
function:The TypeScript code for the JupyterLab extension part is complicated, so it's not clear if what follows is a sufficient summary. Basically, Ajax is used to make a GET request to
/command1
(so Chrome says thatjquery.js
is the initiator for anxhr
type request), while XMLHttpRequest is used to make a POST request to/command2
(so Chrome says thatindex.js
is the initiator for anxhr
type request).The Javascript code of the Ajax request is on line
9600
ofjquery.js
(according to Chrome). The TypeScript code of the XHttpRequest is essentially the following:Failed Attempts to Diagnose: (For the website) JupyterLabHub says the following in the console:
I don't think the hub prefix was manually set in
page_config.json
(I am not the site admin, which makes debugging harder for me) orjupyterhub_config.py
, so is just the default value.Clicking the JupyterLabHub 'Control Panel' button takes the user to
https://hubwebsite/hub/home
. Clicking the JupyterLabHub 'Logout' button (ultimately) takes the user tohttps://hubwebsite/hub/login
. (I am not sure which route/URL it points to initially.)I tried to compare all of this with the JupyterLabHub code and figure out what the values of
hubHost
,hubPrefix
, andbaseUrl
are. Based on the URLs/routes I would have guessedhubHost
=https://hubwebsite/hub/
andhubPrefix
=''
=baseUrl
. But obviously the console says that it must actually behubHost
=https://hubwesite/
andhubPrefix
=hub/
=baseUrl
.When logged in (i.e. at the front page of the JupyterLab part of JupyterLabHub), the URL is
https://hubwebsite/lab/user/krinsman
.Comparing the URL of the login page with the URL of the logout button displayed in the JupyterHub control panel
https://hubwebsite/hub/logout
(not the JupyterLabHub logout button -- I can't figure out which URL that button points to) and with the JupyterHub source code, it seems that JupyterHub thinks the "base route" ishttps://hubwebsite/hub
. (I don't say base URL because I think that is/
, so that the "root web page/route" is "base route" + "base URL", but really the terminology is unclear to me.)On one hand, it seems like JupyterLab (also JupyterLabHub?) thinks that the "base route" is
https://hubwebsite/lab/user
based on the JupyterLab front page URL as well as the route forcommand2
. On the other hand, it seems like the "base route" for JupyterLab ishubwebsite/hub
based on the route forcommand1
.As for what the notebook server extension thinks the "base route" is, or what the value of
nb_server_app.web_app.settings['base_url']
is, I have no idea. (It obviously doesn't agree with what JupyterLab or JupyterLabHub or JupyterHub thinks it is though, or else everything would still be hunky-dory like it is in the SSH port forwarding case.)Even more confusing is that there is a
base_url
orbase_Url
orbaseURL
orbaseUrl
(I don't remember exactly) variable somewhere in the JupyterHub source code too, but I don't understand what it refers to (except that it apparently can't be the same as the base URL of the JupyterLab or of the notebook server or someone else).I couldn't think of anything else to check besides this, and still being confused I figured it would be best to post this issue here, if for no other reason so that it can be documented and indexed by Google.