Extend the plugin system to enable more use cases

[jtpio]

TLJH's focus on simplicity and good defaults is great, which makes it very tempting to use it for other cases than it was designed for!

One of these use cases is being able to use TLJH with containers and the DockerSpawner. This is something that was mentioned in #484 and #524 (and some Discourse topics). The Voila Gallery TLJH plugin is another example that uses DockerSpawner to spawn servers: https://github.com/voila-dashboards/tljh-voila-gallery

As of today, the TLJH installer creates a default user environment, which also installs extensions for JupyterLab and rebuilds the application. These steps increase the time it takes to install TLJH, and could ideally be skipped when the default environment is not needed (using Docker images instead).

The idea would be to extend the existing plugin system to allow more advanced usage, while keeping the same defaults as now.

---

@yuvipanda shared good ideas in this comment on how we could proceed:

I want to balance the possibilities offered by TLJH vs the maintenance cost of complexity. We have a well tested set of hub infrastructure here - Proxy setup, TLS, supervision, etc. This is extremely useful in a number of cases, where we ignore the env created for the user. The installer does a lot of unnecessary work, which is very annoying.

The options here are a specific flag like what this PR does, or moving the entire user env creation step to a plug-in, which can then be replaced. This would require adding newer hooks, maybe even a LoggingConfigurable. This would help us define a nice interface for what a user env provider should do, make it easier to add things like docker based installs. Just skipping env creation makes this a free-for-all, which isn't nice. It will also break TLJH upgrades, since we can't assume anything is idempotent.

So I'd suggest we:

1. Move the current conda based env creation to a plug-in, adding hooks as necessary.
2. Make the current plugin the default, installed via its own package.
3. Write at least one other env creation plugin (docker?) to make sure our interface is general enough
4. Document all this.

What do you think? I can help with code review and discussions, but don't have time to actually write code...

[jtpio]

Here is an example of a plugin that doesn't use the default environment: https://github.com/plasmabio/tljh-repo2docker

Instead it uses repo2docker to build Docker images local to the server. This enables multiple and isolated user environments on the same machine.

The environments can be managed via the UI:

Adding an environment is very similar to what one would do on Binder:

The environments can then be selected on the server options page:

Pretty much like a Binder on a single machine :)

[jtpio]

Maybe there could indeed be a new hook added to spec, something like tljh_user_environment.

It might be possible to use some of the pluggy options for the specs to enable only one result for a particular hook. For example with firstresult: https://pluggy.readthedocs.io/en/latest/#first-result-only

The default user environment plugin shipped with TLJH would define this hook. It would be "overridden" by other plugins that implement the same hook. So only one would be executed.

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/a-tljh-plugin-to-build-user-environments-with-repo2docker/4258/1

[psychemedia]

@jtpio Just picking up on the TLJH+dockerspawner comment, there is a recipe documented here for setting up TLJH to use dockerspawner. I imagine that elements of the tljh_repo2docker might be reusable, although the UI requirements and workflow would be slightly different:

• specify docker image rather than github repo;
• perhaps optionally allow the specification of alternatives to the default DockerHub;
• list images downloaded and "installed" via TLJH admin panel, rather than images built by repo2docker, as images available to TLJH users;