Added "cleanup_all" mode and changed behavior of the "cleanup" mode so that:
"cleanup" is to delete intermediate files only on the marked steps ("keep_intermediates": false);
"cleanup_all: is to delete intermediate files on all steps where applicable except for the marked steps ("keep_intermediates": true).
Decoupled cleanup ("clean" and "clean_all") and execution ("local" etc.) modes to make pipeline actions more predictable for users, i.e. no modules execution in cleanup modes and no files removing in execution modes (except for files produced by TomoBEAR itself in some cases).
Disabled unnecessary pool (re-)opening in cleanup modes.
Disabled re-writing of TIME files when using cleanup modes.
Disabled .setUp() execution for modules when using cleanup modes. Otherwise it will delete folders with data to be preserved (e.g., "particles", "particles_table" and other folders which are not pipeline steps) and will re-create them.
Fixed setting pipeline_ending_step for cleanup modes.
Disabled existing cleanup functionality for steps with important data: "BatchRunTomo", "TemplateMatchingPostProcessing".
Reviewed/fixed/commented cleanup functionality for steps "MotionCor2", "CreateStacks", "DynamoCleanStacks", "GCTFCtfphaseflipCTFCorrection".
Added "cleanup_all" mode and changed behavior of the "cleanup" mode so that:
Decoupled cleanup ("clean" and "clean_all") and execution ("local" etc.) modes to make pipeline actions more predictable for users, i.e. no modules execution in cleanup modes and no files removing in execution modes (except for files produced by TomoBEAR itself in some cases).
Disabled unnecessary pool (re-)opening in cleanup modes. Disabled re-writing of TIME files when using cleanup modes. Disabled .setUp() execution for modules when using cleanup modes. Otherwise it will delete folders with data to be preserved (e.g., "particles", "particles_table" and other folders which are not pipeline steps) and will re-create them. Fixed setting pipeline_ending_step for cleanup modes.
Disabled existing cleanup functionality for steps with important data: "BatchRunTomo", "TemplateMatchingPostProcessing". Reviewed/fixed/commented cleanup functionality for steps "MotionCor2", "CreateStacks", "DynamoCleanStacks", "GCTFCtfphaseflipCTFCorrection".