Closed porban closed 7 years ago
An example of logs can be found here on guillimin: '/gsf-624-aa/data/sz10site/mcicsirp/preproc_20170303/logs/'
Might be linked to the the manager being resubmitted.
Should investigate that and also implement #110
Having a different walltime for manager and worker would also be good.
Likely cause is wall time. The recovery mechanism in psom2 is currently broken.
redundant with #104
After a few hours of successful processing and thousands of jobs done, the pipeline is suddenly reset to 0, i.e. with no jobs done. It might correspond to the end of the wall time, but I'm not 100% sure about this.