Spin up Airflow POC using local KinD cluster

[btylerburton]

User Story

In order to determine the best practice for running Airflow at scale, datagovteam would like to create an Airflow POC using Kubernetes in Docker (KinD).

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• [ ] GIVEN [a contextual precondition] \ [AND optionally another precondition] \ WHEN [a triggering event] happens \ THEN [a verifiable outcome] \ [AND optionally another verifiable outcome]

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• [ ] Install KinD on your local machine
• [ ] Install Airflow Helm Chart
• [ ] Deploy airflow helm release
• [ ] Understand what comes packaged with the helm release
• [ ] Configure ingress-controller for webserver interaction

[nickumia-reisys]

This is the airflow-equivalent work for the datagov-brokerpak-solr section under the "Practical Learning (the Solrcloud example here "should" be very similar to the environment in which airflow would exist.)" chapter in

• https://github.com/GSA/data.gov/issues/4483

This ticket is more-so doing everything by-hand. Follow on work would being converting the "by-hand" work into "terraform" work and then finally into "broker" work.

• "By-Hand": invoking the kubectl and helm commands
• "Terraform": representing everything in terraform code and defining the resources based on the kubernetes and helm providers based on the same logic used in the "by-hand" method.
• "Broker": packaging the terraform code into the CSB and creating resources by using the CSB CLI.