Closed amindadgar closed 1 month ago
The updates to all_discord_guilds_analyzer_etl.py
include refining the Airflow DAG by adding the TriggerRule
import, removing a redundant return statement, and adjusting the task decorator and parameter for enhanced task execution flow.
File | Change Summary |
---|---|
dags/all_discord_guilds_analyzer_etl.py |
- Imported TriggerRule from airflow.utils.trigger_rule - Removed a return statement in discord_etl_raw_data - Modified @task decorator in analyze_discord with trigger_rule=TriggerRule.NONE_SKIPPED - Adjusted parameter in analyze_discord.expand |
sequenceDiagram
participant User
participant Airflow
participant discord_etl_raw_data
participant analyze_discord
User->>Airflow: Trigger DAG
Airflow->>discord_etl_raw_data: Execute Task
discord_etl_raw_data->>Airflow: Task Done (without return)
Airflow->>analyze_discord: Execute Task (TriggerRule.NONE_SKIPPED)
analyze_discord->>Airflow: Task Done
Airflow->>User: DAG Execution Complete
In the land of code, changes sweep, Airflow tasks now run so neat. No returns to halt the way, Trigger rules in full display. Discord data flows with grace, ETL's a smoother race! 🚀🐇
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?
The last task which was analyzer was failing because for one platform data process it was failing and
upstream_failed
was being showned up. Now we've added the trigger ruleNONE_SKIPPED
so it would run if the previous tasks even they were failed.Summary by CodeRabbit
analyze_discord
task runs only if no upstream tasks are skipped.