search

airflow-helm / charts

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
https://github.com/airflow-helm/charts/tree/main/charts/airflow
Apache License 2.0
665 stars 475 forks source link

AWS Secrets manager in helm chart for metadataConnection #831

Open malaa-sa opened 9 months ago

malaa-sa commented 9 months ago

Checks

Chart Version

1.12.0

Kubernetes Version

eks 1.28

Helm Version

version.BuildInfo{Version:"v3.14.0", GitCommit:"3fc9f4b2638e76f26739cd77c7017139be81d0ea", GitTreeState:"clean", GoVersion:"go1.21.6"}

Description

I am trying to find a way to pass RDS credentials to metadataConnection. It works with manual definition. However, when I try to pass the values through external secrets, it doesn't work and results in a podCreation error.

I created the below kubernetes external secret Store:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: tf-eks-airflow-store
  namespace: airflow
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: airflow-worker

External secret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: tf-eks-airflow-db-secret
  namespace: airflow
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: tf-eks-airflow-store
    kind: SecretStore
  target:
    name: tf-eks-airflow-db-secret
    creationPolicy: Owner
  data:
  - secretKey: user
    remoteRef:
      key: tf-eks-airflow-db-secret
      property: db_user
  - secretKey: pass
    remoteRef:
      key: tf-eks-airflow-db-secret
      property: db_password
  - secretKey: protocol
    remoteRef:
      key: tf-eks-airflow-db-secret
      property: db_protocol
  - secretKey: port
    remoteRef:
      key: tf-eks-airflow-db-secret
      property: db_port
  - secretKey: db
    remoteRef:
      key: tf-eks-airflow-db-secret
      property: db_name
  - secretKey: host
    remoteRef:
      key: tf-eks-airflow-db-secret
      property: db_host
  - secretKey: sslmode
    remoteRef:
      key: tf-eks-airflow-db-secret
      property: db_sslmode

Describe:

kubectl describe secret/tf-eks-airflow-db-secret -n airflow                                                                                                                                                             
Name:         tf-disco-eks-airflow-db-secret
Namespace:    airflow
Labels:       reconcile.external-secrets.io/created-by=165bb0f700f243e43dc09fdc3b8c41f4
Annotations:  reconcile.external-secrets.io/data-hash: 46e2369a206811c3a36d3dbeecb5a7d3

Type:  Opaque

Data
====
port:      4 bytes
protocol:  10 bytes
sslmode:   7 bytes
user:      15 bytes
db:        10 bytes
host:      47 bytes
pass:      19 bytes

in values.yaml

  metadataConnection:
    user: airflow_db_user
    protocol: postgresql
    host: disco-app-db.disco-production.svc.cluster.local
    port: 5432
    db: airflow_db
    sslmode: disable
    existingSecret: tf-eks-airflow-db-secret
    pass: pass

I also tried using extraEnvFrom, but also didn't work

extraEnvFrom: |
  - secretRef:
      name: tf-disco-eks-app-api-secret

Relevant Logs

╰─ kubectl get all --namespace=airflow                                                                                                         ─╯
NAME                                       READY   STATUS                            RESTARTS      AGE
pod/airflow-redis-0                        1/1     Running                           0             24h
pod/airflow-run-airflow-migrations-wp88d   0/1     CreateContainerConfigError        0             19m
pod/airflow-scheduler-6b68955d6d-dbf5r     3/3     Running                           0             24h
pod/airflow-scheduler-7597c6ddfd-46dzr     0/3     Init:CreateContainerConfigError   0             19m
pod/airflow-triggerer-0                    0/3     Init:CreateContainerConfigError   0             31m
pod/airflow-webserver-55f6b49599-vldsz     0/1     Init:CreateContainerConfigError   0             19m
pod/airflow-webserver-fcd5bf797-szq52      1/1     Running                           1 (24h ago)   24h
pod/airflow-worker-0                       0/3     Pending                           0             24h

NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/airflow-redis       ClusterIP   172.20.132.34   <none>        6379/TCP   3d23h
service/airflow-triggerer   ClusterIP   None            <none>        8794/TCP   3d23h
service/airflow-webserver   ClusterIP   172.20.138.79   <none>        8080/TCP   3d23h
service/airflow-worker      ClusterIP   None            <none>        8793/TCP   3d23h

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/airflow-scheduler   1/1     1            1           3d23h
deployment.apps/airflow-webserver   1/1     1            1           3d23h

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/airflow-scheduler-6b68955d6d   1         1         1       3d23h
replicaset.apps/airflow-scheduler-6c745966b4   0         0         0       31m
replicaset.apps/airflow-scheduler-7597c6ddfd   1         1         0       19m
replicaset.apps/airflow-scheduler-8699c5bb55   0         0         0       22m
replicaset.apps/airflow-webserver-55f6b49599   1         1         0       19m
replicaset.apps/airflow-webserver-6f9c9d4db4   0         0         0       22m
replicaset.apps/airflow-webserver-d45dcd959    0         0         0       31m
replicaset.apps/airflow-webserver-fcd5bf797    1         1         1       3d23h

NAME                                 READY   AGE
statefulset.apps/airflow-redis       1/1     3d23h
statefulset.apps/airflow-triggerer   0/1     3d23h
statefulset.apps/airflow-worker      0/1     3d23h

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/airflow-run-airflow-migrations   0/1           19m        19m

--

kubectl get secrets,secretstore -n airflow                                                                                                  ─╯
NAME                                               TYPE                 DATA   AGE
secret/airflow-broker-url                          Opaque               1      3d23h
secret/airflow-fernet-key                          Opaque               1      3d23h
secret/airflow-metadata                            Opaque               1      23m
secret/airflow-redis-password                      Opaque               1      3d23h
secret/airflow-ssh-git-key                         Opaque               1      11d
secret/airflow-ssh-git-secret                      Opaque               1      11d
secret/airflow-ssh-git-secrets                     Opaque               1      11d
secret/airflow-webserver-secret-key                Opaque               1      3d23h
secret/letsencrypt-production                      kubernetes.io/tls    2      9d
secret/sh.helm.release.v1.airflow.v1               helm.sh/release.v1   1      3d23h
secret/sh.helm.release.v1.airflow.v2               helm.sh/release.v1   1      32m
secret/sh.helm.release.v1.airflow.v3               helm.sh/release.v1   1      23m
secret/sh.helm.release.v1.airflow.v4               helm.sh/release.v1   1      20m
secret/tf-eks-airflow-db-secret              Opaque               7      66m
secret/tf-eks-airflow-sshkeysecret-secret    Opaque               1      11d
secret/tf-eks-airflow-webserver-secret-key   Opaque               1      5d2h

NAME                                                         AGE   STATUS   CAPABILITIES   READY
secretstore.external-secrets.io/tf-eks-airflow-store   11d   Valid    ReadWrite      True

Custom Helm Values

No response