search

astronomer / dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files
Apache License 2.0
1.21k stars 183 forks source link

Support telemetry during DAG parsing emitting data to Scarf #250

Closed tatiana closed 1 month ago

tatiana commented 1 month ago

Export telemetry related to DAG Factory usage to Scarf.

This data assists the project maintainers in better understanding how DAG Factory is used. Insights from this telemetry are critical for prioritizing patches, minor releases, and security fixes. Additionally, this information supports critical decisions related to the development road map.

Deployments and individual users can opt out of analytics by setting the configuration:

[dag_factory] enable_telemetry False

As described in the official documentation, it is also possible to opt-out by setting one of the following environment variables:

AIRFLOW__DAG_FACTORY__ENABLE_TELEMETRY=False
DO_NOT_TRACK=True
SCARF_NO_ANALYTICS=True

In addition to Scarf's default data collection, DAG Factory collects the following information:

No user-identifiable information (IP included) is stored in Scarf, even though Scarf infers information from the IP, such as location, and stores that. The data collection is GDPR compliant.

The data is not currently being emitted for pre-releases except from integration tests.

The Apache Foundation supports this same strategy in many of its OpenSource projects, including Airflow (#39510).

Example of visualisation of the data via the Scarf UI:

Screenshot 2024-10-17 at 01 56 09 Screenshot 2024-10-17 at 01 55 59 Screenshot 2024-10-17 at 01 55 47 Screenshot 2024-10-17 at 01 55 42

Closes: #214

codecov-commenter commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.04%. Comparing base (6f31846) to head (ad1063a). Report is 1 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #250 +/- ## ========================================== + Coverage 87.71% 89.04% +1.32% ========================================== Files 6 8 +2 Lines 586 648 +62 ========================================== + Hits 514 577 +63 + Misses 72 71 -1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.