Restructure earthmover logs to include log_type - Githubissues

edanalytics / edu_edfi_airflow

Manages extract-load of Ed-Fi data in Airflow

Other

4 stars 0 forks source link

Restructure earthmover logs to include log_type #75

Open gnguyen87 opened 1 week ago

gnguyen87 commented 1 week ago

Description & motivation

This PR adds more metadata to earthbeam logs. It parses through the task_idof each log to return its log_type.

PR Merge Priority:

[x] Low
[ ] Medium
[ ] High

Changes to existing files:

earthbeam_dag.py: the log_to_snowflake() function now has 3 more lines of code to restructure logs.

Tests and QC done

Pending Airflow access