Potentially use qhub for each of the hackweek

[lsetiawan]

Recently I've been playing around with QHub, which will switch name to Nebari soon. This software is open source and a very easy and straightforward way to deploy a jupyterhub ecosystem with batteries included. See below components. To manage resources and deployment you only use a yaml file like so:

project_name: uwhackweek
provider: aws
domain: snowex.uwhackweek.org
certificate:
  type: lets-encrypt
  acme_email: email@example.com
  acme_server: https://acme-v02.api.letsencrypt.org/directory
security:
  authentication:
    type: GitHub
    config:
      client_id: 'someclientid'
      client_secret: 'somesecretkey'
  keycloak:
    initial_root_password: somePasswordForRoot
default_images:
  jupyterhub: quay.io/nebari/nebari-jupyterhub:v0.4.3
  jupyterlab: quay.io/nebari/nebari-jupyterlab:v0.4.3
  dask_worker: quay.io/nebari/nebari-dask-worker:v0.4.3
storage:
  conda_store: 60Gi
  shared_filesystem: 100Gi
theme:
  jupyterhub:
    hub_title: QHub - uwhackweek
    hub_subtitle: Autoscaling Compute Environment on Amazon Web Services
    welcome: Welcome to snowex.uwhackweek.org. It is maintained by <a href="http://quansight.com">Quansight
      staff</a>. The hub's configuration is stored in a github repository based on
      <a href="https://github.com/Quansight/qhub/">https://github.com/Quansight/qhub/</a>.
      To provide feedback and report any technical problems, please use the <a href="https://github.com/Quansight/qhub/issues">github
      issue tracker</a>.
    logo: /hub/custom/images/jupyter_qhub_logo.svg
    primary_color: '#4f4173'
    secondary_color: '#957da6'
    accent_color: '#32C574'
    text_color: '#111111'
    h1_color: '#652e8e'
    h2_color: '#652e8e'
    version: v0.4.4
helm_extensions: []
monitoring:
  enabled: true
argo_workflows:
  enabled: false
kbatch:
  enabled: false
cdsdashboards:
  enabled: true
  cds_hide_user_named_servers: true
  cds_hide_user_dashboard_servers: false
terraform_state:
  type: remote
namespace: snowex
qhub_version: 0.4.4
amazon_web_services:
  region: us-west-2
  kubernetes_version: '1.23'
  node_groups:
    general:
      instance: m5.xlarge
      min_nodes: 1
      max_nodes: 1
    user:
      instance: m5.xlarge
      min_nodes: 1
      max_nodes: 5
    worker:
      instance: m5.xlarge
      min_nodes: 1
      max_nodes: 5
profiles:
  jupyterlab:
  - display_name: Small Instance
    description: Stable environment with 2 cpu / 8 GB ram
    default: true
    kubespawner_override:
      cpu_limit: 2
      cpu_guarantee: 1.5
      mem_limit: 8G
      mem_guarantee: 5G
  - display_name: Medium Instance
    description: Stable environment with 4 cpu / 16 GB ram
    kubespawner_override:
      cpu_limit: 4
      cpu_guarantee: 3
      mem_limit: 16G
      mem_guarantee: 10G
  dask_worker:
    Small Worker:
      worker_cores_limit: 2
      worker_cores: 1.5
      worker_memory_limit: 8G
      worker_memory: 5G
      worker_threads: 2
    Medium Worker:
      worker_cores_limit: 4
      worker_cores: 3
      worker_memory_limit: 16G
      worker_memory: 10G
      worker_threads: 4
environments:
  environment-dask.yaml:
    name: dask
    channels:
    - conda-forge
    dependencies:
    - python
    - ipykernel
    - ipywidgets==7.7.1
    - qhub-dask==0.4.4
    - python-graphviz
    - pyarrow
    - s3fs
    - gcsfs
    - numpy
    - numba
    - pandas
    # - pip:
    #   - kbatch
  environment-dashboard.yaml:
    name: dashboard
    channels:
    - conda-forge
    dependencies:
    - python==3.9.13
    - ipykernel==6.15.1
    - ipywidgets==7.7.1
    - qhub-dask==0.4.4
    - param==1.12.2
    - python-graphviz==0.20.1
    - matplotlib==3.3.2
    - panel==0.13.1
    - voila==0.3.6
    - streamlit==1.10.0
    - dash==2.6.1
    - cdsdashboards-singleuser==0.6.2

Components

The technology stack is an integration of the following existing open source libraries:

• Terraform a tool for building, changing, and versioning infrastructure.
• Kubernetes a cloud-agnostic orchestration system
• Helm: a package manager for Kubernetes
• JupyterHub: a shareable compute platform for data science
• JupyterLab: a web-based interactive development environment for Jupyter Notebooks
• Dask: a scalable and flexible library for parallel computing in Python
  • Dask-Gateway: a secure, multi-tenant server for managing Dask clusters
• Keycloak Open Source Identity and Access Management
• GitHub Actions: a tool to automate, customize, and execute software development workflows in a GitHub repository.
• traefik for routing web/tcp traffic inside cluster
  • traefik-forward-auth single sign on and easy securing of web applications

Amongst the newly created open source libraries on the tech stack are:

• jupyterhub-ssh brings the SSH experience to a modern cluster manager.
• jupyter-videochat allows video-chat with JupyterHub peers inside JupyterLab, powered by Jitsi.
• conda-store serves identical conda environments and controls its life-cycle.
• conda-docker, an extension to the docker concept of having declarative environments that are associated with Docker images allowing tricks and behaviour that otherwise would not be allowed.
• vscode built-in web editor tied to jupyterlab server

Integrations

In an effort for QHub to serve as a core that services can integrate with.

• prefect workflow management
• clearml machine learning platform
• prometheus cluster monitoring
• grafana cluster monitoring visualizations