Only able to download small batches of files at a time

[benkirk]

Hi, thanks for your work on esgpull. Our organization is in the process of transitioning to it from synda and a question has recently come up.

2-3 months ago I was able to successfully use our institution's credentials associated with esgf-node.llnl.gov and downloaded ~6,000 files at a time without issue. We then didn't use the tool for a while. Next time I needed it, we were unable to log in to esgf-node.llnl.gov, I'm guessing that is due to recent authentication changes at the provider, but honestly have no idea... So I created another account with esgf-data.dkrz.de and am generally able to log in, download, etc...

However, when downloading large file counts, I observe the following behavior:

• I can download about ~49 files at a time (not sure why, but this number is consistent and repeatable)
• After that threshold, >95% or so of the download requests fail with a message like

  [2024-03-29 07:55:40]  DEBUG     httpcore.connection 
  connect_tcp.failed exception=ConnectError(OSError('All connection attempts failed'))

• Running esgpull retry && esgpull download is enough to start the process over, then another ~49 successes, and the cycle repeats.

Sorry for the wandering report. It "feels" like the downloads are being throttled by an upstream host or something.

Has anyone seen similar? Are there any workarounds, like throttling the download rate to avoid this behavior, or any other clues?

I'm presently working through a request of ~30K files, and its taking forever 49 at a time.

Thanks in advance for any insights!

-Ben Kirk NCAR Computational & Information Systems Laboratory

[svenrdz]

Hi Ben, thanks for trying out esgpull and for this report.

Yes, I can confirm we have observed a similar issue a few weeks ago at IPSL, with a number of successful downloads about the same as you mention, ~49 files at a time. I have not taken the proper time to investigate this issue yet, since my colleague @AtefBN told me he had this issue mostly with a single query (although a large one too) and found a work around by running esgpull retry && esgpull download in a crontab if I'm not mistaken. I'll try to find the root cause and push a proper fix in the coming days.