Investigate joblib to implement some sort of caching between runs

kaashmonee / rM2FS

Reduction pipeline for the M2FS telescope

0 stars 0 forks source link

Investigate joblib to implement some sort of caching between runs #73

Open kaashmonee opened 5 years ago

kaashmonee commented 5 years ago

Currently, to fit and debug100th spectrum, we have to wait for all the other spectra to be fit, which takes several seconds. However, we want to be able to cache these values so that subsequent runs are faster and we can debug the 100th much more quickly should we need to.

It seems that the joblib could be a good candidate to accomplish this goal.

kaashmonee commented 5 years ago

The current solution, which has been implemented in #76, is to save the pickled files and load them automatically, which seems to do the job. However, if the user quits the program in the middle, the data is not cached, and the next run the program will start over. This should still be implemented so this issue will remain open.