spedas / pyspedas

Python-based Space Physics Environment Data Analysis Software
https://pyspedas.readthedocs.io/
MIT License
143 stars 58 forks source link

Inefficient index searching in SOLO load routine, excessive log output #782

Closed jameswilburlewis closed 4 weeks ago

jameswilburlewis commented 4 months ago

Some of the Solar Orbiter data files at SPDF are organized with timestamps down to the second in the filename, e.g.:

solo_l1_swa-eas-padc_20200620t120002-20200620t130001_v01.cdf

We are currently using 60 sec resolution when generating the dailynames, and trying to match each possible minute of the start time to the links in the directory index. And we get a log message for each attempt that doesn't match (so, up to 1440 log messages per day for the duration of the time interval being loaded). But as far as I can tell, there is generally only a single file per day, it just starts at an unpredictable minute.

We could probably just use a dailynames resolution of 1 day, and wildcard the HHMMSS part of the timestamp, without downloading too much extra data.