Deploy pangeo-forge bakery in USGS cloud

🚀 💥 We successfully ran the pangeo-forge-flink runner integration test on the USGS aws account. There were several bugs along the way, but we did it. These are partially problems with pangeo-forge and its flink-runner Greg (at Development Seed) and Charles (at Columbia) are making strides on these and upcoming PRs should make this all easier and better documented. And part of the problem is getting it all working on USGS cloud. In hindsight the USGS piece was easy, but it's quickly overwhelming

I'll start formulated notes here, which can transition to a more formal document. To the extent practicle, these changes should be incorporated as PRs to pangeo-forge or NASA-IMPACT to minimize the amount of custom USGS documentation.

Deploying the flink runner

install the repo
and awscli v2, kubectl, saml2aws, and (optionally) k9s
create a s3 bucket for storing the terraform state
modify the terraform deployment to pass the correct permission boundary and aws profile (if used).
also modify the deployment to use an existing VPC (It will try to use default, which is disabled on USGS cloud). This VPC does not require an interconnection; however, because everything is orchestrated through kubectl and the AWS API.
terraform init, terraform apply
the eks cluster should deploy but will fail to deploy any of the helm modules
next run aws eks update-kubeconfig --name <cluster-name> --profile <aws profile> --region <aws-region>, which will populated ~/.kube/config with your cluster credentials.
at this point you should be able to interrogate the cluster via kubectl or k9s
add `config_path="~/.kube/config" to the Kubernetes section of the helm provider block
now run terraform apply again to install the helm modules.
after a couple rounds all the modules should install, but you will still get an error related to ingress.
investigating the logs should show that ingress is indeed running; for example, https://kubernetes.github.io/ingress-nginx/troubleshooting/

Installing pangeo-forge-runner

this process can be rough but should improve once some PRs get merged.

create a python environment
(currently) pip install beam==2.52.0
pip install pangeo-forge-recipes
pip install pangeo-forge-runner
to verify, run pangeo-forge-runner --help

Running a recipe

follow the steps in the integration test
modify config.py to point to a preexisting bucket. Also critical that the beam version matches that in our environment
modify the config.py to include aws secret, key, and token from your current session.
run the recipe via the pangeo-forge-runner CLI tool
we can monitor things by running watch -n 1 "kubectl get flinkdeployment,deploy,pod,svc"
or kubectl port-forward --pod-running-timeout=2m0s --address 127.0.0.1 svc/recipe-rest 0:8081 and the opening a browser to 127.0.0.1:XXXX will drop us directly into flink.

Additional links

Next steps

to setup an action, we'll need to verify whether we can saml2aws login via a github action or must this occur via gitlab?
set credential environment variables using $(saml2aws script)
create deploy-run-destroy actions that accepts saml credentials as inputs

hytest-org / hytest