scraper will fail when there are too many erros while retrieving xblocks
Implementation details
there is now a global watcher for all xblocks, with
total_count: the total number of xblocks to retrieve
dl_count: the total number of xblocks download attempt so far
success_count: the total number of xblocks successful download so far
failed_xblocks: details about failed xblocks
there are two parameters to control scrapper stop:
watcher_min_ratio: the minimum ratio of successful downloads (compared to the number of download attempts)
watcher_min_dl_count: the minimum number of xblocks to have attempted to download before stopping the scraper (because otherwise the ratio might be wrong just because we are unlucky in terms of xblocks download order)
Rationale
Fix #160
Changes
Implementation details
there is now a global watcher for all xblocks, with
total_count: the total number of xblocks to retrievedl_count: the total number of xblocks download attempt so farsuccess_count: the total number of xblocks successful download so farfailed_xblocks: details about failed xblocksthere are two parameters to control scrapper stop:
watcher_min_ratio: the minimum ratio of successful downloads (compared to the number of download attempts)watcher_min_dl_count: the minimum number of xblocks to have attempted to download before stopping the scraper (because otherwise the ratio might be wrong just because we are unlucky in terms of xblocks download order)these two parameters can be set at the CLI level