MLflow Import in Airflow DAGs Causes S3 Logging Failure

Checks

[X] I have checked for existing issues.
[X] This report is about the User-Community Airflow Helm Chart.

Chart Version

1.13.1

Kubernetes Version

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.24.7

Helm Version

version.BuildInfo{Version:"v3.13.1", GitCommit:"3547a4b5bf5edb5478ce352e18858d8a552a4110", GitTreeState:"clean", GoVersion:"go1.20.8"}

Description

Description: When attempting to use import mlflow in one of our Airflow DAGs for experiment tracking, we've encountered a critical issue where S3 logging ceases to function. Despite configuring S3 logging in Airflow, no logs are being stored in the designated S3 bucket after adding the MLflow import statement to our DAGs.

Steps to Reproduce:

Create an Airflow DAG with S3 logging configured. Import the mlflow library in the DAG file. Run the DAG and monitor S3 logging behavior.

Expected Behavior: Airflow DAGs should continue logging to the specified S3 bucket even when importing the mlflow library for experiment tracking.

Actual Behavior: After importing mlflow, S3 logging stops functioning, and no logs are stored in the designated S3 bucket.

Impact: This issue severely impacts our ability to track and monitor DAG executions and results in a loss of crucial logging data stored in S3.

Relevant Logs

energy-training-job-train-ek59pn71
*** No logs found on s3 for ti=<TaskInstance: energy_training_job.train manual__2024-04-16T09:13:17.821542+00:00 [success]>
*** Could not read served logs: [Errno -3] Temporary failure in name resolution

Custom Helm Values

config:
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
    AIRFLOW__CORE__LOAD_EXAMPLES: "False"
    AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
    AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://logs/airflow/logs"
    AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "minio_s3_conn"

airflow-helm / charts