My EKS cluster is not connecting with the specified RDS in values.yaml

Siva-Taicho77 commented 11 months ago

Checks

[X] I have checked for existing issues.
[X] This report is about the User-Community Airflow Helm Chart.

Chart Version

8.8.0

Kubernetes Version

Kubernetes version 1.28 Launched with AWS EKS service.

Helm Version

version.BuildInfo{Version:"v3.13.1", GitCommit:"3547a4b5bf5edb5478ce352e18858d8a552a4110", GitTreeState:"clean", GoVersion:"go1.20.8"}

Description

I am trying to setup an airflow application which has been configured to use PostgreSQL RDS for storing metadata. Both the EKS cluster and RDS share the same VPC, the worker node can communicate with RDS. In pod airflow-migrations init contianer check-db is failing (Error logs has been shared in the log section,). Can anyone please let me know why is it trying to reach out to airflow-cluster-pgbouncer.airflow-cluster.svc.cluster.local" (192.168.115.94) instead of the RDS cluster?

Relevant Logs

[2023-12-12T05:58:36.158+0000] {cli_action_loggers.py:135} WARNING - Failed to log action (psycopg2.OperationalError) connection to server at "airflow-cluster-pgbouncer.airflow-cluster.svc.cluster.local" (192.168.115.94), port 6432 failed: FATAL:  database "airflow_cluster1" does not exist
connection to server at "airflow-cluster-pgbouncer.airflow-cluster.svc.cluster.local" (192.168.115.94), port 6432 failed: FATAL:  database "airflow_cluster1" does not exist

(Background on this error at: https://sqlalche.me/e/14/e3q8)
/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/db.py:1740 RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)

Custom Helm Values

########################################
## CONFIG | Airflow Configs
########################################
airflow:
  ## if we use legacy 1.10 airflow commands
  legacyCommands: false

  ## configs for the airflow container image
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/airflow-version.md
  image:
    repository: apache/airflow
    tag: 2.6.3-python3.9

  ## the airflow executor type to use
  executor: KubernetesExecutor

  ## the fernet encryption key (sets `AIRFLOW__CORE__FERNET_KEY`)
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/set-fernet-key.md
  ## [WARNING] change from default value to ensure security
  fernetKey: "7T512UXSSmBOkpWimFHIVb8jK6lfmSAvx4mO6Arehnc="

  ## the secret_key for flask (sets `AIRFLOW__WEBSERVER__SECRET_KEY`)
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/set-webserver-secret-key.md
  ## [WARNING] change from default value to ensure security
  webserverSecretKey: "THIS IS UNSAFE!"

  ## environment variables for airflow configs
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/airflow-configs.md
  config:
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
    AIRFLOW__CORE__LOAD_EXAMPLES: "False"

  ## a list of users to create
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/airflow-users.md
  users:
    - username: admin
      password: admin
      role: Admin
      email: admin@example.com
      firstName: admin
      lastName: admin

  ## a list airflow connections to create
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/airflow-connections.md
  connections: []

  ## a list airflow variables to create
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/airflow-variables.md
  variables: []

  ## a list airflow pools to create
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/airflow-pools.md
  pools: []

  ## extra pip packages to install in airflow Pods
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/extra-python-packages.md
  ## [WARNING] this feature is not recommended for production use, see docs
  extraPipPackages: []

  ## extra environment variables for the airflow Pods
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-environment-variables.md
  extraEnv: []

  ## extra VolumeMounts for the airflow Pods
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
  extraVolumeMounts: []

  ## extra Volumes for the airflow Pods
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
  extraVolumes: []

  ## configs generating the `pod_template.yaml` file for `AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE`
  ## [NOTE] the `dags.gitSync` values will create a git-sync init-container in the pod
  ## [NOTE] the `airflow.extraPipPackages` will NOT be installed
  kubernetesPodTemplate:

    ## the full content of the pod-template file (as a string)
    ## [NOTE] all other `kubernetesPodTemplate.*` are disabled when this is set
    stringOverride: ""

    ## resource requests/limits for the Pod template "base" container
    ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
    resources: {}

    ## extra pip packages to install in the Pod template
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/extra-python-packages.md
    ## [WARNING] this feature is not recommended for production use, see docs
    extraPipPackages: []

    ## extra VolumeMounts for the Pod template
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
    extraVolumeMounts: []

    ## extra Volumes for the Pod template
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
    extraVolumes: []

###################################
## COMPONENT | Airflow Scheduler
###################################
scheduler:
  ## the number of scheduler Pods to run
  replicas: 1

  ## resource requests/limits for the scheduler Pods
  ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
  resources: {}

  ## configs for the log-cleanup sidecar of the scheduler
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/log-cleanup.md
  logCleanup:
    enabled: true
    retentionMinutes: 21600

  ## configs for the scheduler Pods' liveness probe
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/scheduler-liveness-probe.md
  livenessProbe:
    enabled: true

    ## configs for an additional check that ensures tasks are being created by the scheduler
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/scheduler-liveness-probe.md
    taskCreationCheck:
      enabled: false
      thresholdSeconds: 300
      schedulerAgeBeforeCheck: 180

###################################
## COMPONENT | Airflow Webserver
###################################
web:
  ## the number of web Pods to run
  replicas: 1

  ## resource requests/limits for the web Pods
  ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
  resources: {}

  ## configs for the Service of the web Pods
  service:
    type: ClusterIP
    externalPort: 8080

  ## configs generating the `webserver_config.py` file
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/airflow-configs.md#webserver_configpy
  webserverConfig:
    ## the full content of the `webserver_config.py` file (as a string)
    stringOverride: |
      from airflow import configuration as conf
      from flask_appbuilder.security.manager import AUTH_DB

      # the SQLAlchemy connection string
      SQLALCHEMY_DATABASE_URI = conf.get("core", "SQL_ALCHEMY_CONN")

      # use embedded DB for auth
      AUTH_TYPE = AUTH_DB

    ## the name of a Secret containing a `webserver_config.py` key
    existingSecret: ""

###################################
## COMPONENT | Airflow Workers
###################################
workers:
  ## if the airflow workers StatefulSet should be deployed
  enabled: false

###################################
## COMPONENT | Triggerer
###################################
triggerer:
  ## if the airflow triggerer should be deployed
  enabled: true

  ## the number of triggerer Pods to run
  replicas: 1

  ## resource requests/limits for the triggerer Pods
  ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
  resources: {}

  ## maximum number of triggers each triggerer will run at once (sets `AIRFLOW__TRIGGERER__DEFAULT_CAPACITY`)
  capacity: 1000

###################################
## COMPONENT | Flower
###################################
flower:
  ## if the airflow flower UI should be deployed
  enabled: false

###################################
## CONFIG | Airflow Logs
###################################
logs:
  ## the airflow logs folder
  path: /opt/airflow/logs

  ## configs for the logs PVC
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/log-persistence.md
  persistence:
    enabled: false

###################################
## CONFIG | Airflow DAGs
###################################
dags:
  ## the airflow dags folder
  path: /opt/airflow/dags

  ## configs for the dags PVC
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/load-dag-definitions.md
  persistence:
    enabled: false

  ## configs for the git-sync sidecar
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/load-dag-definitions.md
  gitSync:
    enabled: false

###################################
## CONFIG | Kubernetes Ingress
###################################
ingress:
  ## if we should deploy Ingress resources
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/ingress.md
  enabled: false

###################################
## CONFIG | Kubernetes ServiceAccount
###################################
serviceAccount:
  ## if a Kubernetes ServiceAccount is created
  create: true

  ## the name of the ServiceAccount
  name: ""

  ## annotations for the ServiceAccount
  annotations: {}

###################################
## CONFIG | Kubernetes Extra Manifests
###################################

## a list of extra Kubernetes manifests that will be deployed alongside the chart
## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/extra-manifests.md
extraManifests: []

###################################
## DATABASE | PgBouncer
###################################
pgbouncer:
  ## if the pgbouncer Deployment is created
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/pgbouncer.md
  enabled: true

  ## resource requests/limits for the pgbouncer Pods
  ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
  resources: {}

  ## sets pgbouncer config: `auth_type`
  authType: md5

###################################
## DATABASE | Embedded Postgres
###################################
postgresql:
  ## if the `stable/postgresql` chart is used
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/embedded-database.md
  ## [WARNING] the embedded Postgres is NOT SUITABLE for production deployments of Airflow
  ## [WARNING] consider using an external database with `externalDatabase.*`
  enabled: false
###################################
## DATABASE | External Database
###################################
externalDatabase:
  ## the type of external database
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-database.md
  #type: postgres

  host: <redacted>
  port: 5432

  ## the schema which will contain the airflow tables
  database: airflow_cluster1

  ## (username - option 1) a plain-text helm value
  user: postgres

  ## (username - option 2) a Kubernetes secret in your airflow namespace
  #userSecret: "airflow-cluster1-database-credentials"
  #userSecretKey: "username"

  ## (password - option 1) a plain-text helm value
  password: <redacted>

###################################
## DATABASE | Embedded Redis
###################################
redis:
  ## if the `stable/redis` chart is used
  enabled: false

###################################
## DATABASE | External Redis
###################################
externalRedis:
  ## the host of the external redis
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-redis.md
  host: localhost

anirao commented 11 months ago

I am using 8.8.0 and see a similar issue: 2023-12-11T02:24:32.625+0000] {cli_action_loggers.py:135} WARNING - Failed to log action (psycopg2.OperationalError) connection to server at "airflow-pgbouncer.airflow-cluster.svc.cluster.local" (180.64.126.241), port 6432 failed: SSL SYSCALL error: EOF detected (Background on this error at: https://sqlalche.me/e/14/e3q8) /home/airflow/.local/lib/python3.9/site-packages/airflow/utils/db.py:1738 RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9) sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "airflow-pgbouncer.airflow-cluster.svc.cluster.local" (180.64.126.241), port 6432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections?

Siva-Taicho77 commented 11 months ago

@anirao Is there any way to get in touch with the experts here?

thesuperzapper commented 8 months ago

I know it's been a while since this issue was raised, but the issue the original poster @Siva-Taicho77 was having was just that they had not created a database called airflow_cluster1 in their RDS instance.

For the other people like @anirao who are getting sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "airflow-pgbouncer.airflow-cluster.svc.cluster.local" (180.64.126.241), port 6432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections?, you should check the logs for the pgbouncer pod to see if it's running, and can see your RDS instance.

Siva-Taicho77 commented 4 months ago

This issue can be closed, I was able to solve it.

airflow-helm / charts