mcvaneede
commented
9 years ago In general having as many executors as the number of subjects in the pipeline makes most sense. Many parts of the pipeline involve working on a stage for each input file in parallel, after which there is a stage which combines that information before it can continue with each individual file in parallel again.
Given that, the fact that the number of running executors ended up being the number of input files makes sense. When one input file takes a lot longer for a particular (non linear) registration stage than the others, it also makes sense for the remaining executors to die, because the whole pipeline can only continue when that stage finishes (to create the next average to be used as target for the following registration stage for instance).
Generally, when new stages become available, the server process will spawn new executors. So when the longer running registration stage completed, that should have happened. Did you run the check_pipeline_status.py program on the uri to find out what the state of the pipeline was when it was stranded?
----- Original Message -----
From: "Gabriel A. Devenyi" notifications@github.com To: "Mouse-Imaging-Centre/pydpiper" pydpiper@noreply.github.com Sent: Tuesday, November 18, 2014 9:51:52 AM Subject: [pydpiper] BUG: pypdpiper SGE executors die for long processing steps, leave pipeline stranded (#153)
I'm running MBM on a SGE cluster. My run submission was; MBM.py --queue=sge --num-executors=240 --proc=1 --mem=4 --verbose --pipeline-name=rat --lsq6-large-rotations --registration-method=mincANTS --calc-stats --init-model=../MBM-input/model/A10FISP.mnc ../MBM-input/*mnc
The jobs executors seem to get paired down to the number of input files (25) and process away.
For some reason, one of the non-linear registrations too much longer than the others to complete, in that time, it looks like the other executors died off.
When the longer-running registration completed, no new executors were spawned, and the pipeline was left stranded with nothing to do the work.
— Reply to this email directly or view it on GitHub .
gdevenyi
bcdarwin
jasonlerch
I'm running MBM on a SGE cluster. My run submission was;
The jobs executors seem to get paired down to the number of input files (25) and process away.
For some reason, one of the non-linear registrations too much longer than the others to complete, in that time, it looks like the other executors died off.
When the longer-running registration completed, no new executors were spawned, and the pipeline was left stranded with nothing to do the work.