search

getsentry / airflow-metrics

Metrics for airflow
Apache License 2.0
14 stars 6 forks source link

feat: Count tasks in each state #3

Closed Zylphrex closed 5 years ago

Zylphrex commented 5 years ago

To indiciate the number of tasks in each state. For example, a high number of tasks being stuck in the queued/scheduled state can indicate that the system is behaving abnormally.

Zylphrex commented 5 years ago

I did notice one thing when I ran this in composer where the state transition to scheduled weren't reported. This could either have been

  1. I installed the plugin after the state transition happened
  2. The web server (or whichever process it is) isn't running the plugin so it missed the state transition.

I'm going to do some more tests to verify this.

Zylphrex commented 5 years ago

This does do all the counting correctly, however due to Datadog only has 1s of granularity, so it will over write other metrics within the second. In the case of state counting, many of the state transitions happen very close to each other especial when scheduling and they happen in different processes so the metric recorded on Datadog becomes basically useless as some are overwritten.