Adds tasks to harvest the +/- 25 000 URLs of meetings from Cevi and Remmicom.
I tested locally with tasks having 100 URLs running every 5 minutes and it completed smoothly in a day.
I decided, for QA, to go with tasks that contain 50 URLs. We execute one task every 5 minutes. It should take approximately 2 days to complete. Having 5 minutes between each small task leaves room to process other tasks that will arrive to the collector, via scheduled jobs for example.
Relates to OP-1860 To be tested with https://github.com/lblod/harvest-collector-service/pull/12
Adds tasks to harvest the +/- 25 000 URLs of meetings from Cevi and Remmicom.
I tested locally with tasks having 100 URLs running every 5 minutes and it completed smoothly in a day. I decided, for QA, to go with tasks that contain 50 URLs. We execute one task every 5 minutes. It should take approximately 2 days to complete. Having 5 minutes between each small task leaves room to process other tasks that will arrive to the collector, via scheduled jobs for example.