This repository manages the continuous deployment of the Pangeo Cloud Federation JupyterHub Kubernetes clusters using hubploy. It contains scripts to automatically redeploy when the image definition or chart parameters are changed. Changing the image will typically take ~20 minutes, and changing a Helm config variable ~1 minute.
Name | Cloud: region | Staging |
Production |
---|---|---|---|
gcp-uscentral1b | GCP: us-central1-b | https://staging.us-central1-b.gcp.pangeo.io | https://us-central1-b.gcp.pangeo.io |
aws2 | AWS: us-west-2 | https://staging.aws-uswest2.pangeo.io | https://aws-uswest2.pangeo.io |
ooi | Azure: eastus | https://staging.ooi.pangeo.io | https://ooi.pangeo.io |
The first step to using this automation is to create a Kubernetes cluster. Scripts to do so using Google Cloud Platform can be found here. For other cloud providers (e.g. AWS, Azure), follow the Zero-to-JupyterHub guide.
You will need to install
git-crypt
. git-crypt
is used
to encrypt the secrets that are used for deploying your cluster. Please read this HOW GIT-CRYPT WORKS if new to it.
Once you have a cluster created, you can begin customizing the configuration.
git-crypt unlock /path/to/your.key
foobar
as our deployment name from here on).
cp -r example foobar
git-crypt status | grep secrets
deployments/foobar/config
.hubploy.yaml
config file.deployments/foobar/config
.
deployments/foobar/image
.
Error: UPGRADE FAILED: "example.pangeo.io-staging" has no deployed releases
helm delete example-staging --purge
anywhere you have run gcloud container clusters get-credentials
We have some rudimentary testing of deployments. The tests are located in the file test.py
.
Tests should be grouped according to which cloud deployment they should run on (all, GCP only, etc.)
There were some manual setup steps
pangeo-bot
)We've deployed prometheus and grafana to monitor cluster usage. These metrics are publically visible:
These are deployed independently of CI. The GCP deployment is done in deployments/gcp-uscentral1b/Makefile, with the metrics
target.
$ cd deployments/gcp-uscentral1b
$ make metrics
Configuration is in the metrics
folder. We expose it with an nginx-ingress, which was configured to have a static IP on Google Cloud and added to our DNS to server from the URL above.
Likewise with the aws / icesat2 deployment.
All pull requests should be made to the staging
branch. When a change has been verified and is ready for
deployment, staging
can be merged into prod
. This repository is set up with continuous deployment upon pushes
to both staging
and prod
.