Open kaxil opened 1 month ago
Hi @kaxil I'm interested in this issue, may I have a try for this, thanks!
Awesome, assigned it to you
Hi @josix Let me know if you need any help.
also provide a simple grafana dashboard in the example stack ( cause the main airflow grafana dashboard open source is based on statsd ) would help a lot
@howardyoo is also interested in working on this. He led AIP-49, so @josix if you need help please let Howard no.
@howardyoo is equally interested in leading this effort too
Yeah, I just checked out the AIP and the sharing in the Airflow Submit these days, and currently worked on studying the codebase around StatsD and OTel. I believe it would be a better choice for @howardyoo to lead this topic 🙂 Please feel free to unassign me and assign sub-items to me if possible, thanks for the coordination.
I am interested too in this. Please assign me any tasks needed to be done. I am working on similar lines on this in this PR https://github.com/apache/airflow/pull/39908
@howardyoo Could you add a comment, please? GitHub doesn't allow me to assign you an issue until you have commented on it since you aren't part of the Apache org
@howardyoo Could you add a comment, please? GitHub doesn't allow me to assign you an issue until you have commented on it since you aren't part of the Apache org
Ok, done!
Awesome, assigned it to you.
Oh, I guess the above mentioned PR draft got released, which is nice - but looks like it may be on hold for now.
Currently, Apache Airflow uses StatsD for metrics collection and monitoring. To modernize our observability stack and align with industry standards, we should remove StatsD and adopt OpenTelemetry as the primary metrics collection and monitoring tool for Airflow 3. This is made easier with AIP-49 Open Telemetry Support.
Backward Compatibility / Migration
Most of the enterprise Airflow deployments rely on StatsD for monitoring & alerting so we should make sure that there is a smooth migration path. Ideally, all the StatsD metrics mentioned in https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#metric-descriptions should have a like-by-like replacement.
High-level Objectives: