Closed rajiv03 closed 4 years ago
Hi @rajiv03
It sounds like what you need to do is overload the jobs to get high CPU utilization
First, dump the entire set of commands into a file called "joblist" or similar.
Make sure you've set up your QBATCH_* environment variables to define QBATCH_NODES, and QBATCH_PPJ=72 and QBATCH_SYSTEM="pbs"
Then I would do something like
qbatch -c (length of joblist)/5 -j (72*N) joblist
What we need to decide on is how many multiples of 72 i.e. the N value will saturate the CPU. One thing to consider however, is, is your code inefficient in some way? Perhaps you're IO limited and that's why you're not getting much speedup?
Closing as there has been no further response from reporter.
I've huge size(~300000000 million) of list and wants to use a java written algorithm(one executable commnd line) to do some calculation. This algorithm completes the calculation mostly takes less than a min on avg of 1-2 min.
I have been using gnu parallel but jobs are not faster/low cpu utilization. I want to use 5 nodes(each have 72 cpus) or more with pbs torque scheduler in qbatch.
fyi : gnu parallel commands I use
export JOBS_PER_NODE=72 job_parallel="parallel -j $JOBS_PER_NODE --slf $PBS_NODEFILE --wd $WDIR --joblog process.log --resume --progress"
$job_parallel -a input_list.lst sh run_input.sh {}
I would be gratful if you could suggest how do i run effiencely utiizing all the cpus for this large list? Thank you