tropo_pyaps3: parallel downloads

insarlab / MintPy

Miami InSAR time-series software in Python

https://mintpy.readthedocs.io

Other

555 stars 244 forks source link

tropo_pyaps3: parallel downloads #1195

Open ritwika21 opened 1 month ago

ritwika21 commented 1 month ago

Description of proposed changes

Reminders

[ ] Fix #xxxx
[ ] Pass Pre-commit check (green)
[ ] Pass Codacy code review (green)
[ ] Pass Circle CI test (green)
[ ] Make sure that your code follows our style. Use the other functions/files as a basis.
[ ] If modifying functionality, describe changes to function behavior and arguments in a comment below the function declaration.
[ ] If adding new functionality, add a detailed description to the documentation and/or an example.

yunjunz commented 1 month ago

Thank you @ritwika21 for contributing!

Could you add some description to the PR?

And some questions on the ERA5 parallel downloading:

How much time does it save now with this PR, compared with the current version, with an example?
Based on what I was aware of a couple of years ago, ECMWF (via the Copernicus Climate Data Store) allows a max of 3 submitted jobs per user at the same time. Will it make more sense to set the parallel job number to 3, instead of 64, if the 3-job-limit still exists?

yunjunz commented 1 month ago

pre-commit.ci autofix

falkamelung commented 1 month ago

I heard that downloading one date as one file is not the right approach. ECMWF might have options to download all the required days for one SAR dataset as one file. I asked some atmospheric scientists for help, and they were shocked about how we download the data. But I did not pursue this yet.

But until the overall approach is fixed, any improvement is of course greatly appreciated!

yunjunz commented 1 month ago

Any link to the code or documentation for the downloading all-at-once approach will be very helpful.