Pods fail to initialize when using azure blob store for logs

Checks

[X] I have checked for existing issues.
[X] This report is about the User-Community Airflow Helm Chart.

Chart Version

8.8..0

Kubernetes Version

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.10

Helm Version

version.BuildInfo{Version:"v3.13.1", GitCommit:"3547a4b5bf5edb5478ce352e18858d8a552a4110", GitTreeState:"clean", GoVersion:"go1.20.8"}

Description

When I use an azure blob store pvc, check-db on all airflow pods fails and the pods do not run. I can confirm that my pvc configuration is okay because I can use it for non-airflow images, mount volumes and write to it.

Relevant Logs

Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/pathlib.py", line 1323, in mkdir
    self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2023-12-12'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/logging/config.py", line 564, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.9/logging/config.py", line 745, in configure_handler
    result = factory(**kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/log/file_processor_handler.py", line 49, in __init__
    Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.9/pathlib.py", line 1332, in mkdir
    if not exist_ok or not self.is_dir():
  File "/usr/local/lib/python3.9/pathlib.py", line 1439, in is_dir
    return S_ISDIR(self.stat().st_mode)
  File "/usr/local/lib/python3.9/pathlib.py", line 1232, in stat
    return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2023-12-12'

Custom Helm Values

USER-SUPPLIED VALUES:
airflow:
  config:
    AIRFLOW__CORE__LOAD_EXAMPLES: "False"
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "True"
    HTTP_PROXY: <proxy>
    HTTPS_PROXY: <proxy>
  connections: []
  executor: KubernetesExecutor
  extraEnv:
  - name: AIRFLOW__CORE__FERNET_KEY
    valueFrom:
      secretKeyRef:
        key: fernet-key
        name: airflow-secrets
        namespace: airflow-cluster
  - name: AIRFLOW__WEBSERVER__SECRET_KEY
    valueFrom:
      secretKeyRef:
        key: webserver-secret-key
        name: airflow-secrets
        namespace: airflow-cluster
  extraPipPackages: []
  extraVolumeMounts:
  - mountPath: /opt/airflow/cfgdags
    name: dags-volume
  extraVolumes:
  - configMap:
      name: airflow-dags-config
      optional: true
    name: dags-volume
  fernetKey: ""
  image:
    pullPolicy: Always
    pullSecret: <>
    repository: apache-airflow
    tag: 2.6.1-python3.9
  kubernetesPodTemplate:
    extraPipPackages: []
    extraVolumeMounts: []
    extraVolumes: []
    image:
      pullPolicy: Always
      repository: apache-airflow
      tag: 2.6.1-python3.9
    resources: {}
    stringOverride: ""
  legacyCommands: false
  pools: []
  resources: {}
  users:
  - email: admin@example.com
    firstName: admin
    lastName: admin
    password: admin
    role: Admin
    username: admin
  variables: []
  webserverSecretKey: ""
dags:
  gitSync:
    enabled: false
  path: /opt/airflow/cfgdags
  persistence:
    enabled: false
externalDatabase:
  authType: scram-sha-256
  database: airflow_db
  host: ""
  namespace: airflow-cluster
  password: postgres-password
  port: 5432
  serverSSL:
    mode: verify-ca
  type: postgres
  user: postgres-username
externalRedis:
  host: localhost
extraManifests: []
flower:
  enabled: false
ingress:
  enabled: true
  web:
    host: <host>
    ingressClassName: nginx
logs:
  path: /opt/airflow/logs
  persistence:
    enabled: true
    existingClaim: airflow-logs-pvc
pgbouncer:
  enabled: true
  image:
    pullSecret: <>
    repository: airflow-helm/pgbouncer
    tag: 1.18.0-patch.1
  resources: {}
postgresql:
  enabled: true
  image:
    pullPolicy: IfNotPresent
    pullSecrets:
    - <>
    registry: <>
    repository: airflow-helm/postgresql-bitnami
    tag: 11.16-patch.0
  persistence:
    enabled: true
    size: 4Gi
    storageClass: ""
redis:
  enabled: false
scheduler:
  image:
    pullPolicy: Always
    pullSecret: <>
    repository: apache-airflow
    tag: 2.6.1-python3.9
  livenessProbe:
    enabled: true
    taskCreationCheck:
      enabled: false
      schedulerAgeBeforeCheck: 180
      thresholdSeconds: 300
  logCleanup:
    enabled: false
    retentionMinutes: 21600
  replicas: 1
  resources:
    limits:
      cpu: 1000m
      memory: 1G
    requests:
      cpu: 500m
      memory: 512Mi
serviceAccount:
  annotations: {}
  create: true
  name: ""
triggerer:
  capacity: 1000
  enabled: true
  image:
    pullPolicy: Always
    pullSecret: <>
    repository: apache-airflow
    tag: 2.6.1-python3.9
  replicas: 1
  resources:
    limits:
      cpu: 1000m
      memory: 1G
    requests:
      cpu: 500m
      memory: 512Mi
web:
  image:
    pullPolicy: Always
    pullSecret: <>
    repository: apache-airflow
    tag: 2.6.1-python3.9
  replicas: 1
  resources:
    limits:
      cpu: 1000m
      memory: 1G
    requests:
      cpu: 500m
      memory: 512Mi
  service:
    externalPort: 8080
    type: ClusterIP
  webserverConfig:
    existingSecret: ""
    stringOverride: |
      from airflow import configuration as conf
      from flask_appbuilder.security.manager import AUTH_DB

      # the SQLAlchemy connection string
      SQLALCHEMY_DATABASE_URI = conf.get("core", "SQL_ALCHEMY_CONN")

      # use embedded DB for auth
      AUTH_TYPE = AUTH_DB
workers:
  enabled: false
  logCleanupEnabled: false

airflow-helm / charts

Pods fail to initialize when using azure blob store for logs #810