Switch 'launch now' crawl launch process to Airflow crawl tasks

ukwa / ukwa-services

Deployment configuration for all UKWA services stacks.

Apache License 2.0

5 stars 5 forks source link

Switch 'launch now' crawl launch process to Airflow crawl tasks #83

Closed anjackson closed 2 years ago

anjackson commented 2 years ago

Rather than depending on processes running on Ingest, the crawls should be launched from Airflow. First, we'll just port the current mechanism over. (see #??? for planned improvements).

[x] First use the bypm crawl launch as a test case, check it all works fine.
[x] Check those crawls launched properly and down-stream tasks are working as expected.
[x] Then add the npld crawl launch task, working in the same way.

anjackson commented 2 years ago

Done, taking advantage of some downtime to switch over!