sul-dlss / dlme-airflow

This is a new repository to capture the work related to the DLME ETL Pipeline and establish airflow
Apache License 2.0
1 stars 0 forks source link

Refactor post-harvest tasks into filters based on collection config #478

Closed aaron-collier closed 5 months ago

aaron-collier commented 5 months ago

@jacobthill I switched this from a PR against your branch to a new PR that includes your prior work (as this was a branch of your branch) to avoid the need for you to review/merge then me to review/merge the end result. Just removing the duplicate step :-)

This looks like a HUGE pr... it's mostly tests and test fixture contributing the number of files and number of lines added. I may pare that big fixture down to improve the test a bit, but I can do that as a follow up.

That said, it may be worth walking through this as there is the change to the configs.