Mouse-Imaging-Centre / pydpiper

Python code for flexible pipeline control
Other
25 stars 10 forks source link

If `pipeline_executor.py ...` command is too long, it will fail silently #442

Open nzxwang opened 4 years ago

nzxwang commented 4 years ago

The only indication will be in the executor log.

parallel: Error: Command line too long (142448 >= 131049) at input 0: pipeline_executor.py --local --uri-file /hpf/large...

This can occur if a user runs a pipeline with a lot of files. Then the user will just see "submitting executors" while the executors already failed to connect and are no longer in the squeue or qstat.