search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.26k stars 1.42k forks source link

[Datadog] Create out of the box metrics for Dagster internals #12322

Open tha23rd opened 1 year ago

tha23rd commented 1 year ago

What's the use case?

Datadog has an integration with Airflow that provides a ton of out of the box metrics for internal airflow processes: https://docs.datadoghq.com/integrations/airflow/?tab=host

Use case:

It's currently difficult to create monitors on things like:

  1. Daemon performance (all daemons, sensors etc)
  2. Executor queue state
  3. Dagit

Ideas of implementation

The airflow integration is extremely thorough: https://docs.datadoghq.com/integrations/airflow/?tab=host

I don't think something this thorough is needed for a v0, but something to aspire to perhaps!

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

mikals0ft commented 1 year ago

+1 to this. # of queued job runs would definitely be a great metric to have for example.