search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.16k stars 14.04k forks source link

Missing Task Instance StatsD metrics for KubernetesExecutor #11514

Open AlexandreYang opened 3 years ago

AlexandreYang commented 3 years ago

Description

Some StatsD metrics like sent from taskinstance.py:

are not reported when using KubernetesExecutor.

By digging into the code, it seems that methods like TaskInstance._run_raw_task() and TaskInstance.handle_failure() are not reached when using KubernetesExecutor.

Is there a way to make those metrics also available for KubernetesExecutor ?

Use case / motivation

Better observability. Those are useful metrics.

Related Issues

It seems this also apply to CeleryExecutor and possibly other executors too.

CeleryExecutor does report airflow.ti_failures, but not airflow.ti_successes.

boring-cyborg[bot] commented 3 years ago

Thanks for opening your first issue here! Be sure to follow the issue template!

linhnql commented 1 year ago

Hello, can you point me how to fix this, thanks

potiuk commented 1 year ago

There is no solution- someone would need to implement it. I marked it as good first issue and maybe somoene will take care of it if they pick an interest, but otherwise, I think there is no "fix" here. You might also want to attempt it yourself and implement it in your copy of Airlfow and contribute it back to upstream if you are eager to solve it - this is an open source software and anyoe can do it in their copy and upstream back after they test it locally. That would be an awesome first contribution.

linhnql commented 1 year ago

I'm a newbie in Airflow. I use CeleryExecutor for Airflow On the first day, it doesn't report airflow.ti_failures, airflow.ti_successes. But the next day, these metrics appear. I didn't set up, or config anything. I don't know why :(

potiuk commented 1 year ago

Hard to say without seeing any logs or evidences of what happened and knowing your deployment.

kranthikirang commented 1 year ago

we are using kubernetes executor and do not see any issues. However, the DAG image must contain airflow and statd packages. So, that statsd_exporter will do the mapping corectly.

https://github.com/apache/airflow/blob/main/airflow/models/taskinstance.py#L1641

https://github.com/apache/airflow/blob/main/airflow/metrics/statsd_logger.py#L156

ampx-mg commented 4 months ago

@kranthikirang All of our DAG images contain both the packages, but the metrics in the OP are not still being reported