ukwa / ukwa-services

Deployment configuration for all UKWA services stacks.
Apache License 2.0
4 stars 5 forks source link

Simplify and separate W3ACT AirFlow tasks #54

Closed anjackson closed 1 year ago

anjackson commented 2 years ago

Currently, one file contains three workflows, because they share code for dumping the W3ACT DB, and each runs their own dump in case of conflicts due to workflows running simultaniously. To me a bit more canonical-Airflow in style and a bit easier to manage, the workflows could be changed as follows:

This would make it easier to keep them in separate files, which is also more canonical for Airflow, and makes things a bit easier to understand.

anjackson commented 2 years ago

Note in particular that having launch-this-crawl tasks and update-this-access-service tasks tied together is not a great idea. e.g. on DEV when we want to try one without running the other.

anjackson commented 1 year ago

I've separated out the crawl launcher, and I think that's good enough for now. Note that using things like External Task etc. don't really help because it means e.g. the launcher can't run because the other one didn't. Instead, it makes sense for them to be separate and for the launch source files to be atomically updated.