Closed jameswilburlewis closed 6 months ago
This is also happening for Cluster:
>>> wbd_vars = pyspedas.cluster.wbd(trange=['2003-11-01','2003-11-02'],probe=['1','2'])
13-Feb-24 11:31:52: Downloading remote index: https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010000_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010010_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010020_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010030_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010040_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010050_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010100_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010110_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010120_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010130_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010140_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010150_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010200_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010210_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010220_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010230_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010240_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010250_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010300_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010310_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010320_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010330_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010340_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010350_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010400_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010410_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010420_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010430_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010440_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010450_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010500_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010510_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010520_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010530_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010540_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010550_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010600_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010610_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
13-Feb-24 11:31:53: No links matching pattern c1_waveform_wbd_200311010620_v??.cdf found at remote index https://spdf.gsfc.nasa.gov/pub/data/cluster/c1/wbd/2003/11/
Especially in the ground data load routines, the ERG code seems to be making more requests than it needs to get directory indices. Example:
>>> fv=pyspedas.erg.gmag_isee_fluxgate(trange=['2020-08-01','2020-08-02'], site='all')
outputs this:
This is just for the first site, it goes on like this for pages...
I've noticed while one of these routines is running, I am often getting "connection refused" from the server when trying to browse some of these URLs. I think we may be running afoul of some rate-limiting firewall rules. Perhaps our download routine is retrying if it can't retrieve the directory index the first time? The above call used site='all', and we only successfully retrieved data from a single site, probably due to the server blocking so many of the requests.
The erg.load() routine is specifying 'last_version=True' unconditionally when it calls pyspedas.download(), and that is forcing it to fetch directory indices and parse available version numbers, instead of building a static URL for the file to be downloaded. For much of the ground data, I don't think there are even multiple version numbers. (Annoyingly, I can't check the satellite data right now because I'm blocked from the server....)
So maybe we need to change erg.load() and some of the individual load routines to default to version='v01' rather than using 'last_version=True' all the time.
In the past, I've had to disable the akebono test suite on Github because it was failing -- that data, I believe, is also served by the ERGSC, so this might explain why.