Manually spawn jupyter worker

[Hoeze]

Is it possible to manually spawn a worker? E.g. I'd like to run a notebook server on my local desktop, which is not in our Slurm queue.

This way, the Jupyterhub server would work as a proxy for me.

[jbaksta]

I'd be curious to know what you're really attempting to do here. Does your local desktop have direct access to Slurm (or other Scheduler/RM)? You want to use the JupyterHub interface that is attached to the Slurm cluster to spawn instance on your local machine?

[Hoeze]

@jbaksta There is no way to run a worker directly on my desktop:

1. I cannot edit the spawn options directly in the form
2. Even if I could, there would be no way to run that job on my desktop since it's not in the Slurm cluster

Having a single run command for a worker, e.g.: jupyterhub run-worker --remote=https://cluster.com/jupyterhub/ --user=myname --[token=(.*)|ask-pass] would serve two purposes:

• It would allow to manually spawn workers wherever one would like to. Jupyterhub would work as proxy for the workers and nobody has to use stupid port forwardings any more.
• Debugging would probably be easier

[rkdarst]

Currently JupyterHub isn't designed for tasks like this: it intrinsically starts workers and routes to them. This does sound like a useful thing to have, but is almost completely a separate concept from JH (and it doesn't relate to batchspawner directly - it would be better to post to the JH repo).

This could be done by getting the proxy token, and directly communicating with the proxy to add some new routes (e.g. /standalone/your-name ) which are completely separate from the JH routes. The thing to check is that JH may, while updating routes, see these new routes and remove them during its sync time, the hub may need to be modified to only adjust routes in its namespace.

Another idea would be to make a JH external service which does the proxying

... but with all the above, since you're basically making your own thing and JH doesn't provide any real benefit or functionality and it doesn't connect with any of the existing JH code (either server or singleuser-server), one could make their own little service using configurable-http-proxy and do the same as above. Then some Jupyter extension that registers with the proxy. Maybe something already does this?

But... now thinking more of this (just for fun), imagine a JupyterHub spawner that:

• When it spawns, it doesn't actually do anything
• It just emits a messsssage that says "run JUPYTERHUB_API_URL=xxx ... jupyterhub-singleuser --port=xxx ..."
• and listens for the callbacks.
• The spawning process could be automated via the hub API

At least one or two pieces are needed for this, and I'm not sure if there's any benefit... hm...

These are just my first thoughts... may be correct, may not be.

[jbaksta]

I agree that this is interesting in several aspects. However, I'm don't see this directly relating to batchspawner directly, but the core JupyterHub element.

Secondly, there are quite a few security thoughts/hurdles to overcome when thinking about having something spawn things where there may not be good relations, but if were possible, I think the best thing to do is to look into having JupyterHub work the proxy to the known location and that would take some work from what I can see.