search

jupyterhub / wrapspawner

Mechanism for runtime configuration of spawners for JupyterHub
BSD 3-Clause "New" or "Revised" License
60 stars 58 forks source link

Merge profiles into JupyterHub itself #33

Open rkdarst opened 5 years ago

rkdarst commented 5 years ago

I discussed the multiple profile selection tools (ProfileSpawner, the one in KubeSpawner, etc) with @minrk at the Oslo JupyterHub sprint. His proposal was that profiles could be merged directly into JupyterHub since they have been proven by now. This could directly select the spawner class, and avoid all of the proxying problems we have here. Most of the other issues in this repo could be closed, too.

This would need at least the features:

To me the biggest unknown is "how do options forms work", since the options form is a property of the spawner class but there can be multiple spawner classes and one isn't chosen until after the options form is rendered and an option is selected.

What do you all think about this?

rkdarst commented 5 years ago

The master issue will be https://github.com/jupyterhub/jupyterhub/issues/2724, this was just to notify people using wrapspawner and for internal reference.

nikl11 commented 4 years ago

Do I understand it correctly that instead of having only "combination parameters" to choose from a simple list, we could set each option individually? For example similarly to this? Is there no other simple way to be able to use multiple spawners (batchspawner's PBS spawner and kubespawner or local process spawner in my case) and be able to choose the parameters (walltime, ncpus, memory, env variables, image etc.) individually before the notebook server is spawned?

rcthomas commented 4 years ago

You can have both. What we do is extend WrapSpawner itself (we don't use ProfileSpawner), this lets us show a bunch of buttons on /hub/home that users can choose from. Most of these buttons are presets that just spawn a "default" config that 90% of users want 90% of the time. Then there are some other buttons that are just wide open and users can set all the fiddly options they want through the spawner options form for that child spawner.

In our deployment the home/console template page is massively complicated but driven completely by our configuration setup and our customized wrapspawner. We had some opinions that we imposed here (like we want users to have only one server per type and that it should have a specific name that we set).

I don't quite see how profiles/wrapspawner functionality can easily be integrated into JupyterHub itself without a gigantic refactor, so far wrapspawner meets our needs...

nikl11 commented 4 years ago

What you described sounds like exactly I would like to implement in my company. We have a series of clusters connected by a PBS system, and I am in charge of jupyterhub that spawns notebook onto PBS compute nodes using batchspawner's PBSSpawner, as well as using local process spawner (I plan to use kubespawner instead) to spawn notebooks inside the virtual cloud kubernetes cluster where the hub itself runs. Right now I have spawning options set exactly as the wrapspawner's example, just a simple list of pbs parameter combinations (1CPU + 1GB RAM +1 hour, 4 CPUs + 4GB RAM + 8 hours, etc.). This is quite limiting because users would like to set each parameter separately.

Would you be so king and provide me with some examples or source codes just to know where to start and what is or is not possible? A screenshot of everything you are able to set would be also nice.

I agree that merging this with jupyterhub is a loads of work, in my opinion each spawner should provide its own template for setting spawner settings, similarly to what kubespawner offers, and jupyterhub should maybe only choose the spawner similarly to the wrapspawner manner.