Open guynir42 opened 1 year ago
In many cases you'd want to first download all raw data and then run a separate reduction (and analysis) campaign.
The reduction can be done by the observatory, so that part can happen in the same multithreaded block where the download happens.
This saves time because the data is already in memory and doesn't need to be re-loaded from disk.
On the other hand, it is python code, not I/O bound code, so it will just slow down the downloading of files.
Therefore it should be an option to do this at the same time, but not a default.
In many cases you'd want to first download all raw data and then run a separate reduction (and analysis) campaign.
The reduction can be done by the observatory, so that part can happen in the same multithreaded block where the download happens.
This saves time because the data is already in memory and doesn't need to be re-loaded from disk.
On the other hand, it is python code, not I/O bound code, so it will just slow down the downloading of files.
Therefore it should be an option to do this at the same time, but not a default.