The web-scraper will need to run on a configured schedule (e.g. nightly), so we will need a component that can automatically call it on a configurable basis.
Requirements:
Once started, continue running until stopped
Load schedule basis from configuration. (e.g. Every night at 2am, Every 3 days at 4am, Every 6 hours) Only one schedule is needed, but we need flexibility in how it is configured. See "Crontab" for examples.
If custom-written component created (e.g. no cloud-native solution readily available), include documentation on deployment and configuration.
Note: Assuming this is cloud hosted, it is possible that there may already be a cloud-native solution available.
The web-scraper will need to run on a configured schedule (e.g. nightly), so we will need a component that can automatically call it on a configurable basis.
Requirements:
Note: Assuming this is cloud hosted, it is possible that there may already be a cloud-native solution available.