At the moment this is quite a manual process. There is a tactical need for improvement (eg. set up a cron job etc) but also this is the right time to think about architecture (eg how do the scraping pipelines not step on the toes of the data.json harvester?) and what Infra options we have for go-live.
Acceptance criteria:
[x] We have a spec for architecture and DevOps set up internally and an estimate of how long that would take to implement the proposed solution (there could be a tactical and long-term solution)
At the moment this is quite a manual process. There is a tactical need for improvement (eg. set up a cron job etc) but also this is the right time to think about architecture (eg how do the scraping pipelines not step on the toes of the data.json harvester?) and what Infra options we have for go-live.
Acceptance criteria: