how do to parallel-process files with looper locally -- lump them together, this will group several commands in one like:

[donaldcampbelljr]

how do to parallel-process files with looper locally -> originally a divvy idea. issue: 100 files -> divvy submits to cluster no problem, if local they will run serial. Could run in background process using ampersand. in command shell script with &. So can we lump 100 samples (in 10 background processes), new divvy template to accomplish that

• Original Notes

[donaldcampbelljr]

Currently, if using lump for samples locally (--lump-n), we get a couple submission scripts:

#!/bin/bash

echo 'Compute node:' `hostname`
echo 'Start time:' `date +'%Y-%m-%d %T'`

{
python3 /tmp/tmplr4qcfsa/advanced/pipeline/pipeline1.py --sample-name sample1 --req-attr val 
python3 /tmp/tmplr4qcfsa/advanced/pipeline/pipeline1.py --sample-name sample2 --req-attr val 
} | tee /tmp/tmplr4qcfsa/advanced/results/submission/PIPELINE1_lump1.log

#!/bin/bash

echo 'Compute node:' `hostname`
echo 'Start time:' `date +'%Y-%m-%d %T'`

{
python3 /tmp/tmplr4qcfsa/advanced/pipeline/pipeline1.py --sample-name sample3 --req-attr val 
} | tee /tmp/tmplr4qcfsa/advanced/results/submission/PIPELINE1_lump3.log

Perhaps we can change the template to use & for running in parallel?

#!/bin/bash

echo 'Compute node:' `hostname`
echo 'Start time:' `date +'%Y-%m-%d %T'`

{
python3 /tmp/tmplr4qcfsa/advanced/pipeline/pipeline1.py --sample-name sample1 --req-attr val &
python3 /tmp/tmplr4qcfsa/advanced/pipeline/pipeline1.py --sample-name sample2 --req-attr val &
python3 /tmp/tmplr4qcfsa/advanced/pipeline/pipeline1.py --sample-name sample3 --req-attr val & 
} | tee /tmp/tmplr4qcfsa/advanced/results/submission/PIPELINE1_lump1.log