Set up cloud Pangeo/nebari instance

[jonmjoyce]

Create a sandbox environment in AWS following instructions from Pangeo.

[rsignell-usgs]

@jonmjoyce, I'm a big fan of Nebari (formerly called Qhub), open-source Infrastructure-as-Code that uses Terraform to deploy JupyterHub with Dask Gateway on Kubernetes on AWS, Google, Azure or DigitalOcean.

A few of the things that make Nebari attractive over Dask Hub:

• The awesome conda-store approach for managing conda environments. Unlike other approaches, you don't need to custom containers with the environments -- notebooks and each Dask worker get's it's environment from the conda-store pod -- which means that you can update an environment in less than 5 minutes, instead of 30+ minutes with other approaches. And you avoid package version mismatches since notebook and workers are accessing the same environments from conda-store.
• Groups! Users are members of one or more groups, and have the ability to share both code and modify/share environments with other group members
• Easy maintenance via github actions. If you want to change the hub, say, adding a new server type with more memory, you can just edit a config file on github, and when merged, the changes are automatically deployed.

We've been running Nebari for ESIP for the last two years at https://jupyter.qhub.esipfed.org.
I've given you access if you want to check it out! There is an introductory notebook at https://jupyter.qhub.esipfed.org/user/rsignell-usgs/lab/tree/shared/users/Welcome.ipynb

[rsignell-usgs]

@jonmjoyce I saw this commit: https://github.com/asascience-open/nextgen-dmac/commit/1fe08eb5a838ee9c74fbd67f0269f7d5ba24d5cf which seems to suggest you guys are still working on deploying Kubernetes?

What is the reason to do that instead of deploying kubernetes (and JupyterHub + Dask Gageway) using qhub/nebari?

[jonmjoyce]

Appreciate that you are keeping up-to-date with the changes on GitHub! This particular commit you referenced was from some ongoing work I was doing. Prior to our meeting I had already configured a K8s cluster and tested a Pangeo install (single user JupyterHub + dask), and I have since changed that configuration to be a dask-gateway install. Since our meeting, @cheryldmorse is setting up nebari and this ticket is now tracking that progress.

I think Kubernetes is going to be part of our stack going forward regardless. Aside from the analysis capabilities we'll get from nebari, I would like our service mesh to target K8s as a platform. This includes the data ingest workflows that @benjwadams is exploring and the data access protocols that @mpiannucci is working on. In some cases we will also want a headless dask cluster for automated processing and that's another reason for running dask-gateway.

[rsignell-usgs]

@jonmjoyce , I absolutely agree that Kubernetes will likely going to be part of the will be part of the stack, but Nebari is a painless way to configure, deploy and maintain your Kubernetes cluster! I know, because I did it the hard way for two years before switching to Qhub/Nebari!

And you get JupyterHub & Dask Gateway as a bonus!

@cheryldmorse please reach out if you have issues deploying Nebari !!