JupyterHub integration

[kallewesterling]

Assigning @lukehare — let me know if I should assign someone else.

[thobson88]

We're currently hoping to meet on Tue 4th or Wed 5th Oct, with Fede who also has experience deploying JupyterHub.

The aim will be to:

• determine whether the existing JupyterHub instance lwm-jupyterhub can be re-used.
• find out whether Fede deployed another instance that might be more suitable
• learn how to:
  • restart JupyterHub
  • update the configuration
  • test that it's doing what we want

[kallewesterling]

@thobson88 has the date been set? Wed 5th?

[thobson88]

Tentatively set for Tue 4th at 14:30.

And since we have one or two existing JupyterHub instances in the project we'll have this first pass among ourselves, before bringing in Callum if necessary.

[griff-rees]

AFAIK: currently set for Tue 4th at 15:00

Suggestions:

• Deciding between https://tljh.jupyter.org/en/latest/topic/whentouse.html#topic-whentouse
• The Littlest: https://tljh.jupyter.org/en/latest/#the-littlest-jupyterhub
  • 1-100 users
  • easiest to deploy
  • what @fedenanni has used
• Kubernetes: https://z2jh.jupyter.org/en/stable/#zero-to-jupyterhub-with-kubernetes
  • Modular (Docker container for each component)
  • Scalable (if we would ever like to make aspects of this public after LwM)
  • Combine with the Django deploy (Django another Docker image)
• I've been assuming we'd go for the latter (Kubernetes) via Dockerfiles https://z2jh.jupyter.org/en/latest/jupyterhub/customizing/user-environment.html#customize-an-existing-docker-image
  • Postgres
    • At least for Newspaper records (SQL side of Django ORM)
    • Potentially for JupyterHub users as well (or two separate DB Docker images...)
  • Django
    • ORM layer in python
    • manages DB migrations
    • Infrastructure for any public availability in future
    • Manage updates to django/backend
  • JupyterLab
    • Copy of ORM code (or run within the Django layer)
    • Separate users from Django admin users etc.
    • Persistence: managing/saving user Jupyter notebooks
    • Maybe easier to manage JupyterLab upgrades in new images/releases (need to know)
    • Implications for use beyond this current project end date...
  • If Littlest
  • Probably still need to have a copy of ORM layer code
  • Probably still requires a separate Django Docker image

[griff-rees]

Also questions of authetication across all of this. There's storing user accounts in SQL and admin stuff on servers. I've used dotenv.

[kallewesterling]

For documentation: meeting set to 15.30-16.30 today to talk running metadata DB + JupyterHub via Docker.

[griff-rees]

General update: the basic configuration for this is commented out in local.yml. Assuming the server tests go similar to testing locally, reenabling this would be the next step, followed by registering with GitHub for authentication.

[kallewesterling]

I'd like to see us getting this set up as a prototype/MVP if we want to get this rolling. How can we try that, @griff-rees ? As far as I can tell, we'd need to:

1. Set up a test VM
2. Clone the repo
3. Fix the local.yml file's configuration (on the VM? in the repo?)
4. (Set up a metadata db postgres server instance --> this is obviously a separate task, and may be a lot of work)
5. Run the docker-compose on the VM
6. ...does it need anything else?

[griff-rees]

I've renamed this given we had that meeting, and some of this is currently with local.yml as you refer to @kallewesterling (thanks for that comment). Heavily relates to #71