[bug] not create s3 logs & *** Log file does not exist: /opt/airflow/logs

[seunggabi]

Checks

• [X] I have checked for existing issues.
• [X] This report is about the User-Community Airflow Helm Chart.

Chart Version

8.6.1

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:11Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"darwin/arm64"}

Helm Version

version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.19.3"}

Description

1. run dags
2. logs error
3. stop job

How can I fix?

Relevant Logs

*** Falling back to local log
*** Log file does not exist: /opt/airflow/logs/dag_id=asdf/run_id=manual__2023-04-12T13:20:07.505970+00:00/task_id=init/attempt=3.log
*** Fetching from: http://:8793/log/dag_id=asdf/run_id=manual__2023-04-12T13:20:07.505970+00:00/task_id=init/attempt=3.log
*** Failed to fetch log file from worker. Request URL is missing an 'http://' or 'https://' protocol.

### Custom Helm Values

```yaml
airflow:
  airflow:
    securityContext:
      fsGroup: 65534
    image:
      repository: apache/airflow
      tag: 2.4.3-python3.8
    executor: CeleryExecutor
    fernetKey: 7T512UXSSmBOkpWimFHIVb8jK6lfmSAvx4mO6Arehnc=
    webserverSecretKey: "THIS IS UNSAFE!"
    extraPipPackages:
      - awscli
      - apache-airflow
      - apache-airflow-providers-amazon==1.1.0
      - cached_property
    config:
      AIRFLOW__CORE__LOAD_EXAMPLES: false
      AIRFLOW__CORE__REMOTE_LOGGING: "True"
      AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://bucket/airflow/logs"
      AIRFLOW__CORE__REMOTE_LOG_CONN_ID: "aws_default"
      AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
      AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://bucket/airflow/logs"
      AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "aws_default"
      AIRFLOW__WEBSERVER__EXPOSE_CONFIG: false
      AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE: Asia/Seoul
      AIRFLOW__WEBSERVER__BASE_URL: https://airflow.com
  postgresql:
    enabled: false
  redis:
    enabled: false
  externalDatabase:
    type: postgres
    host: "host..."
    port: 5432
    database: airflow
    userSecret: "secret-airflow"
    userSecretKey: "POSTGRES_USER"
    passwordSecret: "secret-airflow"
    passwordSecretKey: "POSTGRES_PASSWORD"
    properties: ""
  externalRedis:
    host: "host..."
    port: 6379
    databaseNumber: 1
    password: ""
    passwordSecret: "secret-airflow"
    passwordSecretKey: "REDIS_PASSWORD"
    properties: ""
  web:
    securityContext:
      fsGroup: 65534
    replicas: 1
    service:
      type: NodePort
  dags:
    gitSync:
      enabled: true
      repo: "git..."
      branch: main
      revision: HEAD
      syncWait: 60
  scheduler:
    securityContext:
      fsGroup: 65534
  workers:
    securityContext:
      fsGroup: 65534

[thesuperzapper]

@seunggabi First, you probably should not be using airflow.extraPipPackages, as it also looks like you are installing packages that are already included in the official airflow image (which includes all airflow providers by default).

Second, if you are wanting to persist your logs on S3, please first try the instructions in our docs, and report back if that is successful.

[seunggabi]

Thanks to fast reply. I read a documents, but I couldn't..

airflow:
  airflow:
    image:
      repository: apache/airflow
      tag: 2.4.3-python3.8
    executor: CeleryExecutor
    fernetKey: 7T512UXSSmBOkpWimFHIVb8jK6lfmSAvx4mO6Arehnc=
    webserverSecretKey: THIS IS UNSAFE!
    config:
      AIRFLOW__CORE__LOAD_EXAMPLES: false
      AIRFLOW__CORE__REMOTE_LOGGING: true
      AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: {s3}
      AIRFLOW__CORE__REMOTE_LOG_CONN_ID: aws_default
      AIRFLOW__LOGGING__REMOTE_LOGGING: true
      AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: {s3}
      AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: aws_default
      AIRFLOW__WEBSERVER__EXPOSE_CONFIG: false
      AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE: Asia/Seoul
      AIRFLOW__WEBSERVER__BASE_URL: https://airflow.com
    extraEnv:
      - name: AWS_DEFAULT_REGION
        value: ap-northeast-2
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: admatch-secret-stg
            key: AWS_ACCESS_KEY_ID
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: admatch-secret-stg
            key: AWS_SECRET_ACCESS_KEY
    securityContext:
      fsGroup: 65534
  postgresql:
    enabled: false
  redis:
    enabled: false
  externalDatabase:
    type: postgres
    host: {host_postgresql}
    port: 5432
    database: airflow
    userSecret: {secret}
    userSecretKey: POSTGRES_USER
    passwordSecret: {secret}
    passwordSecretKey: POSTGRES_PASSWORD
    properties:
  externalRedis:
    host: {host_redis}
    port: 6379
    databaseNumber: 1
    password:
    passwordSecret: {secret}
    passwordSecretKey: REDIS_PASSWORD
    properties:
  web:
    securityContext:
      fsGroup: 65534
    replicas: 1
    service:
      type: NodePort
  dags:
    gitSync:
      enabled: true
      repo: {git}
      branch: main
      revision: HEAD
      syncWait: 60
  scheduler:
    logCleanup:
      enabled: false
    securityContext:
      fsGroup: 65534
  workers:
    logCleanup:
      enabled: false
    securityContext:
      fsGroup: 65534
  logs:
    path: /opt/airflow/logs
    persistence:
      enabled: true
      storageClass: efs-gp
      size: 5Gi
      accessMode: ReadWriteMany

I removed extraPipPackages, but ... I fronted this problem.. ModuleNotFoundError: No module named 'airflow.providers.amazon.aws.operators.emr_add_steps'

---

I set logs conf, I can't read /opt/airflow/logs

---

Please help me .. TT. I wait for comment. thx

[seunggabi]

k8s executor is successed. so I closed this issue.