crunchdao / crunch-cli

7 stars 4 forks source link

Data: Add retry mechanism when downloading the data #95

Closed Caceresenzo closed 3 months ago

Caceresenzo commented 4 months ago

When downloading from AWS with signed URLs, because the file can be very large, users on a small connection could crash and lose all progress.

After re-querying the signed URL and using the range header, the file was able to continue downloading.