Open goktugkose opened 1 month ago
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
cc: @VladaZakharova and Google team
cc: @VladaZakharova and Google team
Sure, we will take a look :)
Actually the BackgroundThreadTransport mode of StackdriverTaskHandler is not working properly.
I also have a airflow running KubernetesExecutor and eventually I have to implement custom logging config to override BackgroundThreadTransport with SyncTransport in order to show full log.
Also showing logs from stackdriver is slow.
Apache Airflow version
2.9.1
If "Other Airflow 2 version" selected, which one?
No response
What happened?
We are experiencing an issue with Stackdriver Logging with Airflow. As changelog suggests, with Airflow 2.9.1 version Google provider is updated to 10.17.0 and solves the bugs with Stackdriver logging. We tested with the latest version. However, we cannot configure remote logging with Stackdriver. As an initial step, we created a log bucket in Cloud Monitoring and also created a log sink to investigate the logs. We use the configuration below in Helm chart,
GOOGLE_APPLICATION_CREDENTIALS
is set as an environment variable and a Google Cloud connection containing the same service account is added to Airflow with these scopeshttps://www.googleapis.com/auth/cloud-platform
,https://www.googleapis.com/auth/logging.admin
Findings:
Problems faced:
Example DAG File:
Produced Logs (not real time, they appear after task completion):
What you think should happen instead?
As suggested by Airflow documentation, logs should be loaded in real time. Other logging configurations such as Elasticsearch provides this behavior. Also, task logs should also be served in the UI. I have tested with GCS Logging and we expect to see the similar logs with Stackdriver logging.
How to reproduce
GOOGLE_APPLICATION_CREDENTIALS
with rolesroles/logging.admin
androles/monitoring.admin
Helm 3.12.3
apache-airflow-providers-google==10.17.0
Operating System
Debian GNU/Linux 12 (bookworm)
Versions of Apache Airflow Providers
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct