Deploy OpenMetaData On Our Shiny New Data Platfork EKS Cluster

mitodl / ol-infrastructure

Infrastructure automation code for use by MIT Open Learning

BSD 3-Clause "New" or "Revised" License

44 stars 4 forks source link

Deploy OpenMetaData On Our Shiny New Data Platfork EKS Cluster #2589

Open feoh opened 1 month ago

feoh commented 1 month ago

Description/Context

We want to add OpenMetaData to our data platform so we can have a single source of truth index for all our data.

We also want to standardize on Kubernetes / EKS for our future deployments.

Plan/Design

Spin up a throw-away K3s cluster for testing this
Write Pulumi code to build an OMD specific Opensearch cluster
Write Pulumi code to create an OMD specific RDS Postgres instance
Deploy OpenMetaData to the K3s cluster using its Helm chart.

feoh commented 2 weeks ago

Override the storage class to use EFS:

helm upgrade openmetadata-dependencies open-metadata/openmetadata-dependencies -n omd  --set "airflow.dags.persistence.storageClass=efs-sc" --set "airflow.logs.persistence.storageClass=efs-sc"

feoh commented 5 days ago

Mike has broken the back of this task. See https://github.com/mitodl/hq/discussions/5463