Increase Airflow logging coverage, review existing logs and add missing essential logs - Githubissues

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

https://airflow.apache.org/

Apache License 2.0

37.12k stars 14.31k forks source link

Increase Airflow logging coverage, review existing logs and add missing essential logs #43180

Open omkar-foss opened 4 weeks ago

omkar-foss commented 4 weeks ago

Description

As per users' feedback in the Airflow Debugging Survey 2024, about 42% of respondents mentioned that logs are missing or non-existent in some cases.

Use case/motivation

Goal for this issue is to review the existing Airflow logs coverage and enhance it by adding logs wherever appropriately required.

Related issues

Parent Issue: https://github.com/apache/airflow/issues/40975

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[x] I agree to follow this project's Code of Conduct

Dev-iL commented 2 weeks ago

A good example of an issue with logging is when (fatal) errors from workers are not being forwarded to the scheduler, and it's very hard to trace what's actually wrong. In cases when a worker is exhibiting an early enough error (being unable to load the full Airflow machinery that records logs locally and forwards them to the scheduler/webserver), displaying a stack trace captured on the worker would help immensely.

Ref: https://github.com/apache/airflow/issues/42136