spedas / pyspedas

Python-based Space Physics Environment Data Analysis Software
https://pyspedas.readthedocs.io/
MIT License
147 stars 58 forks source link

Cluster load_csa routine improvement #582

Open NikGrimm opened 1 year ago

NikGrimm commented 1 year ago

Hi, I have been working with the load_csa routine for a while know. This routine always downloads data from the archive even if the time range was loaded before. This seems to be not really necessary. For other satellites like Themis or MMS the load routine checks if data is already download before it loads new data. Would it be possible to also implement a similar check for teh load_csa routine?

jameswilburlewis commented 1 year ago

Hi NikGrimm,

The load_csa routine works a little differently than most of the other missions we support. That data provider does not store data in CDF format; instead, it generates CDFs on-the-fly from the native storage format every time it's requested, and transfers the data in an archive file that must be unpacked by the client to get at the CDFs. So we can't rely on comparing file modification times on the server and local files to determine if a download is necessary.

With that said -- the CDF files do include some timestamp information in the filenames, so it should be possible to implement a "no_download" option to load already downloaded files from the local data directory. It may take some time (months rather than weeks) to get this feature implemented, but it would be a nice feature to have (in both pyspedas and IDL SPEDAS), so thanks for the suggestion!