For very large emulators, it can sometimes still take many hours (or days, if they get extremely large) to analyze an iteration.
Losing progress in this analysis process due to an interruption (like, timeout on a cluster) could cost a lot of time.
This should be avoided.
I could modify the _evaluate_sam_set method to automatically make a backup of all evaluation progress whenever the entire plausible sample set has been evaluated in a single iteration.
I do not think it can be done any earlier due to that requiring many MPI calls back-and-forth, which heavily slows down the process.
If this modification could be made to specifically work for analyze, the method can check if an analysis process was in progress and continue if so.
I will have to think about what a KeyboardInterrupt does (do I see that as the user not wanting the process to finish and thus PRISM should remove the progress backup?).
For very large emulators, it can sometimes still take many hours (or days, if they get extremely large) to analyze an iteration. Losing progress in this analysis process due to an interruption (like, timeout on a cluster) could cost a lot of time. This should be avoided.
I could modify the
_evaluate_sam_setmethod to automatically make a backup of all evaluation progress whenever the entire plausible sample set has been evaluated in a single iteration. I do not think it can be done any earlier due to that requiring many MPI calls back-and-forth, which heavily slows down the process. If this modification could be made to specifically work foranalyze, the method can check if an analysis process was in progress and continue if so. I will have to think about what aKeyboardInterruptdoes (do I see that as the user not wanting the process to finish and thus PRISM should remove the progress backup?).