Open benkirk opened 4 months ago
Hi Ben, thanks for trying out esgpull
and for this report.
Yes, I can confirm we have observed a similar issue a few weeks ago at IPSL, with a number of successful downloads about the same as you mention, ~49 files at a time.
I have not taken the proper time to investigate this issue yet, since my colleague @AtefBN told me he had this issue mostly with a single query (although a large one too) and found a work around by running esgpull retry && esgpull download
in a crontab if I'm not mistaken.
I'll try to find the root cause and push a proper fix in the coming days.
Hi, thanks for your work on
esgpull
. Our organization is in the process of transitioning to it fromsynda
and a question has recently come up.2-3 months ago I was able to successfully use our institution's credentials associated with
esgf-node.llnl.gov
and downloaded ~6,000 files at a time without issue. We then didn't use the tool for a while. Next time I needed it, we were unable to log in toesgf-node.llnl.gov
, I'm guessing that is due to recent authentication changes at the provider, but honestly have no idea... So I created another account withesgf-data.dkrz.de
and am generally able to log in, download, etc...However, when downloading large file counts, I observe the following behavior:
esgpull retry && esgpull download
is enough to start the process over, then another ~49 successes, and the cycle repeats.Sorry for the wandering report. It "feels" like the downloads are being throttled by an upstream host or something.
Has anyone seen similar? Are there any workarounds, like throttling the download rate to avoid this behavior, or any other clues?
I'm presently working through a request of ~30K files, and its taking forever 49 at a time.
Thanks in advance for any insights!
-Ben Kirk NCAR Computational & Information Systems Laboratory