Operation not permitted: '/opt/airflow/logs/...'

obeyda commented 1 year ago

Checks

[X] I have checked for existing issues.
[X] This report is about the User-Community Airflow Helm Chart.

Chart Version

8.6.1

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:58:16Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.6", GitCommit:"08d3594304660f86cfbd17bbb862041b4b75fe6c", GitTreeState:"clean", BuildDate:"2023-02-08T17:22:59Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}

Helm Version

version.BuildInfo{Version:"v3.7.1", GitCommit:"1d11fcb5d3f3bf00dbe6fe31b8412839a96b3dc4", GitTreeState:"clean", GoVersion:"go1.16.9"}

Description

persistence enabled
used airflow.defaultSecurityContext.fsGroup: 0 without setting any other securityContext
followed the "how to persist logs" doc

result: No problem in airflow pods but KubernetesExecutorPods can't write to the logs volume

Relevant Logs

2023-04-05T02:46:39.148471134Z stderr F PermissionError: [Errno 1] Operation not permitted: '/opt/airflow/logs/dag_id=xxx/run_id=manual__2023-04-05T02:46:26.041418+00:00/task_id=xxx_01'

Custom Helm Values

airflow:
  image:
    tag: 2.5.3-python3.10
  executor: "KubernetesExecutor"
  usersUpdate: false
  config:
    AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "False"
    AIRFLOW__WEBSERVER__COOKIE_SAMESITE: "Lax"
    AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX: "True"
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "True"
  defaultSecurityContext:
    fsGroup: 0
  kubernetesPodTemplate:
    podLabels:
      my-label: "true"
logs:
  path: /opt/airflow/logs
  persistence:
    enabled: true
    storageClass: azurefile
    size: 128Gi
scheduler:
  logCleanup:
    enabled: false
workers:
  enabled: false
redis:
  enabled: false
flower:
  enabled: false
dags:
  gitSync:
    enabled: true
    repo: xxxx
    branch: main
    revision: HEAD
    depth: 1
    repoSubPath: ""
    httpSecret: airflow-git-credentials

thesuperzapper commented 1 year ago

@obeyda this failure is caused by a bug in airflow itself that was introduced in version 2.5.1 (and is still present in 2.5.2, and 2.5.3), see here https://github.com/apache/airflow/issues/29112. The problem is that Azure File does not allow chmod, and airflow mistakenly tries to run chmod on log files if they are not already 777.

For now, you might want to do one of the following:

use another method of log persistence, for example, the Azure Blob Storage remote provider
use airflow 2.5.0 (but this is an older version, which might have separate problems).
use airflow 2.4.3 (but downgrading your DB might be problematic).

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had activity in 60 days. It will be closed in 7 days if no further activity occurs.

Thank you for your contributions.

Issues never become stale if any of the following is true:

they are added to a Project
they are added to a Milestone
they have the lifecycle/frozen label

thesuperzapper commented 1 year ago

Closing because this is an upstream issue, your best bet is to NOT use Airflow 2.5.1, 2.5.2, or 2.5.3 (which all have this bug).

airflow-helm / charts