Bug: Retrieve Images Function Halts on Size Limit Exceedance

[2320sharon]

Bug: Retrieve Images Function Halts on Request Size Limit Exceedance

Description The retrieve images function exhibits a bug where if the request size limit is exceeded more than 10 times, all subsequent downloads are halted. This is not the desired behavior.

Expected Behavior Instead of halting all downloads upon exceeding the size limit, the function should:

• Print the image that caused the size limit to be exceeded.
• Continue with the remaining downloads without interruption.
• Only halt all downloads if there are 10 or more requests that fail to connect to the server.

The rationale is to prioritize the continuation of downloads to ensure the maximum amount of data is retrieved, even in the presence of errors.

Steps to Reproduce

1. Invoke the retrieve images function with a dataset large enough to trigger the size limit exceedance more than 10 times.
2. Observe that after the 10th exceedance, the download process halts.

Possible Solution Implement a check to distinguish between size limit exceedance and connection failures. Ensure that the download process continues despite size limit errors, and reserve the halting mechanism for server connection issues only.

Additional Context The goal of the retrieve images function is to download as much data as possible, hence it's critical to handle errors in a way that minimizes the impact on the download process.

[2320sharon]

1. The retrieve_images function is now more resilient to download errors, tolerating up to 10 download failures without stopping the entire download process.
2. Images that exceed the maximum size, which often indicates corruption, are now detected and skipped instead of being downloaded as unusable pixelated files.
3. The enhanced error handling prevents corrupted images from halting the download process, a problem in the standard Coastsat code has.
4. A logging system has been implemented to record the details of any download issues, including the IDs of files that fail to download and reasons for any metadata file issues.
5. A new test script, test_download_tif.py, has been added to verify the improved error handling with corrupted image downloads. This script can be quickly run locally to verify that corrupted images raise the correct error message in the download_tif function.

Overall, the download function is more stable, especially useful for areas with sparse data.