airflow-helm / charts

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
https://github.com/airflow-helm/charts/tree/main/charts/airflow
Apache License 2.0
630 stars 474 forks source link

Change in metadataSecretName does not redeploy scheduler #826

Closed chillinger closed 2 months ago

chillinger commented 4 months ago

Checks

Chart Version

1.11.0

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.9", GitCommit:"d15213f69952c79b317e635abff6ff4ec81475f8", GitTreeState:"archive", BuildDate:"2023-12-20T00:00:00Z", GoVersion:"go1.21.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.6-gke.1289000", GitCommit:"a53df8308e54c8cea2e8cf44be1842106e93d04f", GitTreeState:"clean", BuildDate:"2024-01-26T09:25:53Z", GoVersion:"go1.20.13 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}

Helm Version

3.11.1

Description

I had a working Airflow deployment, connected to a CloudSQL Instance. I then created a new CloudSQL Instance, restored a Backup into this new instance and changed the Kubernetes Secret with the Connection string to point to this new Instance. The webserver correctly picked up the new secret, but the scheduler was not restarted with the new config. In the Webserver under Admin -> Configs I could see the new DB Connection being present.

Only after manually deleting the scheduler pod and re-applying the helm chart the scheduler was re-created and started with the correct DB connection.

Relevant Logs

No response

Custom Helm Values

airflow:
  executor: KubernetesExecutor
  ingress:
    web:
      enabled: True
      ingressClassName: nginx
      hosts:
        - name: airflow.OUR.TLD
          tls:
            enabled: true
            secretName: tls-cert
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt
  postgresql:
    enabled: false
  data:
    metadataSecretName: connection-string
  networkPolicies:
    enabled: True
  config:
    scheduler:
      min_file_process_interval: 10
    webserver:
      expose_config: True
    providers.jdbc:
      allow_driver_class_in_extra: True
      allow_driver_path_in_extra: True
  extraEnv: |
    - name: AIRFLOW__CORE__LOAD_EXAMPLES
      value: 'True'
chillinger commented 4 months ago

Update: the triggerer Pod is also not redeployed

thesuperzapper commented 2 months ago

This issue is related to the official chart, not the community chart, please raise it on the airflow/airflow repo.