S3 uploads seem to fail very, very rarely in a way that the boto library doesn't seem to want to retry. Given that all the processing is done by the time we upload, it seems like a shame to rely on the queue to retry the whole job. Instead, just loop a few times with exponential backoff to see if we can store the tile.
Also, fix typo / keyword error with the functionality to skip existing downloads.
rmarianski
commented
8 years ago
S3 uploads seem to fail very, very rarely in a way that the
botolibrary doesn't seem to want to retry. Given that all the processing is done by the time we upload, it seems like a shame to rely on the queue to retry the whole job. Instead, just loop a few times with exponential backoff to see if we can store the tile.Also, fix typo / keyword error with the functionality to skip existing downloads.
@rmarianski could you review, please?