search

CS-SI / eodag

Earth Observation Data Access Gateway
https://eodag.readthedocs.io
Apache License 2.0
308 stars 39 forks source link

Mass download issue with creodias and sobloo #437

Open dpriskinGEO opened 2 years ago

dpriskinGEO commented 2 years ago

Describe the bug When I try to download a lot of Sentinel-2 products (around 2000), at some point the EODAG breaks. The code worked perfectly for months.

Output

requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://auth.creodias.eu/auth/realms/dias/protocol/openid-connect/token

Environment:

sbrunato commented 2 years ago

Hello @dpriskinGEO , and thanks for reporting this issue. Could you give us more details on how you perform search & download ? Using several search() or a search_all, downloading using download() in a loop, or download_all ? with some particular parameters ? Thanks

dpriskinGEO commented 2 years ago

Using several search() or a search_all, downloading using download() in a loop, or download_all ? with some particular parameters ?

Hello @sbrunato! I use search_all() and download_all() functions

sbrunato commented 2 years ago

thanks @dpriskinGEO , can you also post the full traceback returned by eodag when it breaks ?

dpriskinGEO commented 2 years ago

@sbrunato 

Traceback (most recent call last):
  File "/home/dp/miniconda3/envs/dag310/lib/python3.10/site-packages/eodag/plugins/authentication/keycloak.py", line 55, in authenticate
    response.raise_for_status()
  File "/home/dp/miniconda3/envs/dag310/lib/python3.10/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://auth.creodias.eu/auth/realms/dias/protocol/openid-connect/token

I could not reproduce the 504 error one yet, but it stops the same way but different error code.