UnixJunkie / PAR

Execute shell commands in a parallel and/or distributed way
https://savannah.nongnu.org/projects/par
GNU General Public License v3.0
2 stars 4 forks source link
bag-of-tasks command-line-tool distributed-computing parallel-computing parameter-sweep python-program

If you use this software, please cite the corresponding publication:

@article{Berenger2010, author = {Berenger, Francois and Coti, Camille and Zhang, Kam Y. J.}, title = {{PAR: A PARallel And Distributed Job Crusher}}, doi = {10.1093/bioinformatics/btq542}, year = {2010}, journal = {Bioinformatics} }

you need to install Pyro, cf. http://pyro.sourceforge.net

(sudo yum install pyro OR sudo aptitude install pyro)

On the server, TCP port 7766 must be open

TCP port 9090 must also be open if the server is also

the Pyro nameserver (default case)

if Pyro is not installed system wide, try this:

export PYTHONPATH=~/usr/Pyro-3.10/build/lib

Some usage examples:

run in parallel commands from test_parallel.input

cat test_parallel.input | ./parallel.py -i /dev/stdin

same than before but with a progress bar

cat test_parallel.input | ./parallel.py -i /dev/stdin -v

run in parallel commands from test_parallel.input and store output

in output.log

./parallel.py -i test_parallel.input -o output.log

same than before but with a user defined output printer

./parallel.py -i test_parallel.input -o output.log -p post_proc_example

real world usage example

1) server side

./parallel.py -v -i many_commands.sh -o par_many_commands.log -s

2) client side, on each machine you want to join the computation

replace SERVER_NAME by the machine name from where you launched

parallel.py using -s

./parallel.py -c SERVER_NAME

3) be thrilled! ;)