Handling of massive data requests

[chad-earthscope]

ROVER should be able to handle massive data requests as gracefully as possible.

To target goals (as extreme end-members):

• Request and download all TA data (e.g. 2004 - now), e.g.:

  rover retrieve TA_*_*_* 2004-1-1 2020-1-1

• Request and download all _GSN data (1970 - now), e.g.:

  $ cat > req.txt
  _GSN * * * 1970-1-1 2020-1-1
  $ rover list-retrieve req.txt

Note: initial evaluation should be done with list-retrive to avoid making actual data requests.

First step is to evaluate where the current bottlenecks are for handling these requests. Then we can determine how to address them.