Open consideRatio opened 3 years ago
Thanks @consideRatio for the detailed breakdown :+1:
a change in JupyterHub's script entrypoint jupyterhub-singleuser has changed between between 1.3.0 and 1.4.1?
Out of curiosity, do you have a link to a diff or PR related to that change? Thanks!
Also this link share extension was initially made to share links to test RTC on the dev Binder for JupyterLab, which starts a separate Jupyter Server with jupyter-server-proxy:
In that case when running on Binder, there are 2 running servers:
The extension should list both, but the end user needs the token from the (first) front server to access the instance.
Thanks @jtpio for your input! I've made no progress since last but is not done trying.
Out of curiosity, do you have a link to a diff or PR related to that change? Thanks!
These are two PRs that I was thinking could potentially have something to do with this. But I'm not sure at all.
https://github.com/jupyterhub/jupyterhub/pull/3437 https://github.com/jupyterhub/jupyterhub/pull/3347
Just a quick note for context - in a conversation with @ellisonbg about this, he suggested something might be happening by virtue of the Github authenticator we're using where this problem manifests, that has to monkeypatch the base authentication classes in JupyterHub, and changes/eliminates the token generation. Binder doesn't use Github auth, so that difference might be the source of the problem.
Not 100% sure it is, but I wanted to capture Brian's point here in case it helps tracking down the problem...
@yuvipanda might also have some thoughts here perhaps?
For reference, these are logs when starting up a user server that doesn't have a token set. My guess is that we observe a difference because of the difference of using SingleuserNotebookApp vs NotebookApp (Or ServerApp).
An important clue to guess this for me has been that @fperez clarified that mybinder.org doesn't start user pods with jupyterhub-singleuser
which mean they will run NotebookApp rather than SingleUserNotebookApp, and hence they may not experience the same thing even though they are started by a JupyterHub.
[I 2021-06-14 11:17:01.004 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.
Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler)
[I 2021-06-14 11:17:10.925 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.1
[I 2021-06-14 11:17:10.933 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: /home/jovyan
[I 2021-06-14 11:17:10.933 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.3.0 is running at:
[I 2021-06-14 11:17:10.933 SingleUserNotebookApp notebookapp:2302] http://jupyter-someuser:8888/user/someuser/
[I 2021-06-14 11:17:10.933 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-06-14 11:17:10.954 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds
So, what is the difference between SingleUserNotebookApp and NotebookApp then? I'm not sure, here are some traces of code that may be relevant.
@consideRatio did you manage to make it work in the end? I'm using the latest (1.1.1) zero-to-jupyterhub-k8s helm chart and trying to figure out how to make it work with the newly released RTC in jupyterhub.
The extension indeed returns a link with no token and I could not manage to share any notebook with user is not allowed
in the logs.
not sure if this helps any..
if you comment out token=""
in the SingleUserNotebookAppMixin class
I'm now getting a token set when viewing
jupyterlab_link_share/servers
however the jupyterhub frontend does not seem to respect or know the token so it does not allow it to bypass auth.
I should add to get that token showing up I first login to jupyterhub with the token=''
in the SingleUserNotebookAppMixin class
then I jupyterlab > hub contol > stop my server
.. then change source so remarked #token=''
in the SingleUserNotebookAppMixin
and then start the user server again without logging out.
If I have token=''
not set in SingleUserNotebookAppMixin
im getting an error on login. The error could be the hub proxy not including the token in the url.
In JupyterHub, you can just set the token to be the value of the env variable JUPYTERHUB_API_TOKEN
, and that works. Gives you full control over the hub until the original user restarts it.
I've updated to 0.2.3, but I'm still getting a link with an empty token in a JupyterHub (1.5.0) context. Is there anything else I need to do?
@yuvipanda you got this to work. What versions of jupyterlab and jupyterhub was used?
Note that its also relevant to know the version of jupyterhub used in the singleuser-server in this case I think.
jovyan@jupyter-yuvipanda:~$ pip list | grep -P 'jupyter|notebook'
jupyter-client 6.1.12
jupyter-contrib-core 0.3.3
jupyter-contrib-nbextensions 0.5.1
jupyter-core 4.9.1
jupyter-highlight-selected-word 0.2.0
jupyter-latex-envs 1.4.6
jupyter-nbextensions-configurator 0.4.1
jupyter-resource-usage 0.6.0
jupyter-rsession-proxy 1.4
jupyter-server 1.11.2
jupyter-server-proxy 3.1.0
jupyter-shiny-proxy 1.1
jupyter-telemetry 0.1.0
jupyter-tree-download 1.0.1
jupyterhub 1.5.0
jupyterlab 3.2.4
jupyterlab-geojson 3.1.2
jupyterlab-link-share 0.2.3
jupyterlab-pygments 0.1.2
jupyterlab-server 2.8.2
jupyterlab-widgets 1.0.2
notebook 6.4.5
is the versions on the user container image. I'm running JupyterHub 1.5 on the hub image as well
Thanks for those details, @yuvipanda! Here's the diff with the same command on my server:
--- yuvipanda-deps.txt 2021-11-22 16:53:28.306201127 +0100
+++ dlukes-deps.txt 2021-11-22 16:53:48.021584625 +0100
@@ -1,22 +1,16 @@
-jupyter-client 6.1.12
-jupyter-contrib-core 0.3.3
-jupyter-contrib-nbextensions 0.5.1
+jupyter 1.0.0
+jupyter-archive 3.2.1
+jupyter-client 7.0.6
+jupyter-console 6.4.0
jupyter-core 4.9.1
-jupyter-highlight-selected-word 0.2.0
-jupyter-latex-envs 1.4.6
-jupyter-nbextensions-configurator 0.4.1
-jupyter-resource-usage 0.6.0
-jupyter-rsession-proxy 1.4
jupyter-server 1.11.2
jupyter-server-proxy 3.1.0
-jupyter-shiny-proxy 1.1
jupyter-telemetry 0.1.0
-jupyter-tree-download 1.0.1
jupyterhub 1.5.0
+jupyterhub-systemdspawner 0.15.0
jupyterlab 3.2.4
-jupyterlab-geojson 3.1.2
jupyterlab-link-share 0.2.3
jupyterlab-pygments 0.1.2
jupyterlab-server 2.8.2
jupyterlab-widgets 1.0.2
-notebook 6.4.5
+notebook 6.4.6
I'm guessing the problem might be that I'm using SystemdSpawner, whereas you're using Docker? The rest of the differences either look like non-essential packages, or I have newer versions.
I'm guessing if I whitelist the token env var in c.Spawner.env_keep
, it might work? Trouble is, I don't know that env var's name. I'll try to figure it out by inspecting JupyterHub's environment, but if someone knows, a hint would be appreciated :)
JUPYTERHUB_API_TOKEN
? (Can't test right away as I currently have users logged in.)
the API token env var is to test manually, the lab extension doesn't need that. I'm not sure what could be happening. If you view source on your jupyterlab page, is there a json blob in the HTML (page config)? If so, does it have a key for hubUser
? That's what is used by #21 to determine if it's on a hub, and maybe it isn't being set? That's what @minrk is referring to
Thanks for the pointers! Well hubUser
is a non-empty string set to my username, so that part is alright and the condition used by #21 to determine whether it's on a hub succeeds. However, the token is then fetched by calling PageConfig.getToken
, which -- AFAICS -- tries two places to get the token: that JSON blob you mentioned, using the key "token"
, and document.body.dataset
, using the key "jupyterApiToken"
. In my case, the first one is set to an empty string, and the second one is undefined.
So whitelisting JUPYTERHUB_API_TOKEN
in my jupyterhub_config.py
doesn't change anything, the token in the generated link is still empty. But I can at least access it in a notebook via os.environ
and manually paste it into the link, which works.
@dlukes ah, you're probably running into the case @minrk mentioned in https://github.com/jupyterlab-contrib/jupyterlab-link-share/pull/21#issuecomment-974010489
Oh, OK, thanks :) So it looks like I might get it to work if I downgrade JupyterHub to <1.5? But it also sounds like the entire token/permissions system is still in flux. I think I might hold off on link sharing for a bit more in that case, it's not such a core piece of functionality for me, and revisit it once it settles down a bit.
@dlukes I am super curious why it doesn't work for you, while it does work for me even though we have very similar hub versions...
Me too! I realize you're also running JupyterHub 1.5. The big difference seems to be SystemdSpawner vs. Docker; I thought whitelisting JUPYTERHUB_API_TOKEN
might do it, but it didn't. I'm all out of ideas on what to investigate further at the moment, but if you have any suggestions, let me know, I'm happy to try them when I get a chance :)
The token may be available if you run a notebook
-based server in jupyterhub 1.5, while a jupyter-server
-based server should (correctly) not put $JUPYTERHUB_API_TOKEN in PageConfig.token.
could you be more specific what you mean with "notebook-based server" and "jupyter-server-based server"
Ah, sorry. There are currently two implementations of the jupyter server. The old one, in the notebook
package, and a new one, in the jupyter-server
package. JupyterHub's single-user server is a 'wrapper' of one of these to integrate authentication with JupyterHub. The behavior is almost the same, but there can be subtle differences (and of course, new features in the new server not available in the old one).
In JupyterHub 1.x, the default behavior of jupyterhub-singleuser
is to extend notebook
(this changes to jupyter-server
in 2.0). If you launch jupyter-labhub
as the single-user entrypoint (e.g. c.Spawner.cmd = 'jupyter-labhub'
), it will always be jupyter-server
.
In both 1.x and 2.x, you can specify which to use with the environment variable:
# notebook, default in 1.x
JUPYTERHUB_SINGLEUSER_APP="notebook.notebookapp.NotebookApp"
# jupyter_server, default in 2.0
JUPYTERHUB_SINGLEUSER_APP="jupyter_server.serverapp.ServerApp"
In JupyterHub 1.x, the default behavior of
jupyterhub-singleuser
is to extendnotebook
(this changes tojupyter-server
in 2.0). If you launchjupyter-labhub
as the single-user entrypoint (e.g.c.Spawner.cmd = 'jupyter-labhub'
), it will always bejupyter-server
.
@yuvipanda Maybe that's the reason your setup works while mine doesn't? I'm using labhub.
@dlukes that seems the most likely explanation
This isn't actually fixed.
I also have this issue. We are running jupyterhub ((it is the littlest jupyterhub setup) with Google authentication, and when I click the share button it leads to a link like this with no token: https://jupyterhub-dev.cheme.cmu.edu/user/jkitchin@andrew.cmu.edu/lab?token=
In a terminal, there is a $JUPYTERHUB_API_TOKEN, which works fine if I add it to that url.
Is there some setting that will put hte token in the URL?
I think this may not be an issue with this extension, but it may influence people using it. Anyhow, I figure I can write down notes while debugging this here.
The issue
I tried to use this extension on my recently deployed JupyterHub (v1.4.1) with quite recent versions of most things, including
jupyterlab==3.1.0a10
. It works visually and so, but a web response from/jupyterlab_link_share/servers
results in a token set to an empty string which in turn results in a link that doesn't include a token.Why?
/jupyterlab_link_share/servers
return an empty string token for me on hub.jupytearth.org while it returns a token on mybinder.org? No clue.What code base is responsible for returning a response when visiting
/jupyterlab_link_share/servers
? It seems like this code:https://github.com/jupyterlab-contrib/jupyterlab-link-share/blob/a1bf019e10d8e548b64ffaf809555946639efbaa/jupyterlab_link_share/handlers.py#L6-L17
That code references notebookapp list_running_servers... Have that function changes between versions? No it had not.
I wonder if list_running_servers have started returning different details because of a change in JupyterHub's script entrypoint
jupyterhub-singleuser
has changed between between 1.3.0 and 1.4.1?Hmmmm it seems like
jupyter --runtime-dir
returns a runtime directory path, and within that path I findnbserver-<pid>.json
with details representing the JSON structure returned.What sets this file, and why is it set without a token or not?
Differences observed
jupyter --version
differencesjupyter --version
and miscJSON Response from /server at mybinder.org
When pressing the share button I observed a request like this https://hub.gke2.mybinder.org/user/jupyterlab-cont-rlab-link-share-ati8onmk/jupyterlab_link_share/servers?1622512360400 result in the following response.
JSON Response from /server at hub.jupytearth.org
When pressing the share button I observed a request like this https://hub.jupytearth.org/user/consideratio/jupyterlab_link_share/servers?1622512267115 result in the following response.
ps aux
at mybinder.orgps aux
at hub.jupytearth.org