The-Academic-Observatory / oaebu-workflows

Telescopes, Workflows and Data Services for the 'Book Analytics Dashboard Project (2022-2025)', building upon the project 'Developing a Pilot Data Trust for Open Access eBook Usage (2020-2022)'
https://documentation.book-analytics.org/
Apache License 2.0
5 stars 0 forks source link

Onix wf custom schedule #244

Closed keegansmith21 closed 3 months ago

keegansmith21 commented 3 months ago

Create a custom Airflow timetable for the Onix Workflow. This makes the workflow run every week on Monday and also on the 5th of every month.

This required the creation of the class, which inherits from Airflow's Timetable base class. We add in the logic for start and end date determination for both scheduled and manual runs. We are also required to register the timetable as a plugin. We do this with the plugin class and also adding the classes to the plugins folder. Weirdly, any DAG that uses the plugin MUST import it relative to the plugin folder. This means that the plugin folder itself must be added to the PYTHONPATH. See this issue for a full explanation.

Testing this with the sandbox environment required an update to the sandbox's create_dag_run function as it was unable to handle custom timetables. See this PR. Naturally, this means that this PR is dependent on the observatory platform's PR.

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 93.94%. Comparing base (be36e85) to head (8bfccf1).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #244 +/- ## ======================================= Coverage 93.94% 93.94% ======================================= Files 15 15 Lines 3038 3040 +2 Branches 389 389 ======================================= + Hits 2854 2856 +2 Misses 102 102 Partials 82 82 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.