search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.51k stars 14.14k forks source link

Unable to see dag_id and task_id tagging for mentioned metrics #39522

Closed EMT-Swaroop closed 3 months ago

EMT-Swaroop commented 4 months ago

Apache Airflow version

2.9.1

If "Other Airflow 2 version" selected, which one?

2.9.0

What happened?

According to airflow documentation, from 2.8.2 airflow version the metrics ti_failures, ti_successes, zombie_killed metrics need to have tagging task_id and dag_id image

Airflow documentation:https://airflow.apache.org/docs/apache-airflow/2.9.1/administration-and-deployment/logging-monitoring/metrics.html

In our airflow environment we have upgraded airflow 2.9.1 and helm chart 1.13.1. Still we are not able to see the tagging for setting up alerts and monitoring at task level.

What you think should happen instead?

The tagging of task-id and dag_id for metrics like ti_failures, ti_successes, zombie_killed need to be available so that we can locate the tasks accordingly to debug and we can have the alerting according to that.

How to reproduce

If we use 2.8.1 or later version tagging need to be there for mentioned metrics. But those are not coming, only the counter values are getting produced by the airflow statsd.

Operating System

Debian Bullseye

Versions of Apache Airflow Providers

recently we have upgraded to 2.9.1.

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 4 months ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

RNHTTR commented 4 months ago

@dirrao can you take a look?

Also, it would probably be a good idea to add to the documentation what metrics tagging even is

EMT-Swaroop commented 4 months ago

Hey @RNHTTR, Below are the list of few metrics that are missing tagging which were mentioned in the airflow documentation which I have mentioned in the issue description.

ti_failures ti_successes zombie_killed

dirrao commented 4 months ago

@RNHTTR , @EMT-Swaroop I will look into this issue.

dirrao commented 4 months ago

@EMT-Swaroop If you are using statsd with influx backend store, you need to enable the influxdb_tags_enabled flag to propagate the tags. If not, Can you share the details about the metrics exporter?

EMT-Swaroop commented 4 months ago

Hey @dirrao , We are not using influxdb, we are using prometheus as backend store and didn't customize any configuration in the statsd deployment, we are using the default stasd exporter which is coming from the parent airflow chart (1.13.1)

EMT-Swaroop commented 4 months ago

@dirrao any update?

EMT-Swaroop commented 4 months ago

@dirrao , kindly help with the issue as we are facing for monitoring the airlfow task fails due to the absence of task_ids

github-actions[bot] commented 3 months ago

This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

github-actions[bot] commented 3 months ago

This issue has been closed because it has not received response from the issue author.

lane-100 commented 2 months ago

@EMT-Swaroop FWIW, it looks like the problem is caused by what's described in this issue, which hasn't (yet) been automatically closed: https://github.com/apache/airflow/issues/40027

lucasfcnunes commented 1 month ago

I'm looking forward to see ti_failures and ti_successes with the appropriate dag_id and task_id labels.