We're likely to hit rate-limiting issues from upstream, or random 503 errors, if we try to do large batches of operations. These should be fairly rare, as the file_hash field in the WOF meta CSV should limit the changes, but sometimes it still happens that a lot of hashes change at the same time. We can probably spread out the impact and reduce the chance of failures by limiting each batch of downloads to a certain number of neighbourhoods each run.
nvkelso
commented
6 years ago
We're likely to hit rate-limiting issues from upstream, or random 503 errors, if we try to do large batches of operations. These should be fairly rare, as the
file_hashfield in the WOF meta CSV should limit the changes, but sometimes it still happens that a lot of hashes change at the same time. We can probably spread out the impact and reduce the chance of failures by limiting each batch of downloads to a certain number of neighbourhoods each run.