Torque compatibility

[jrkirk61]

Hi,

Our group would like to use clusterflow, however, our cluster uses Torque. Will there be a torque-friendly clusterflow in the near future?

Thanks

[ewels]

Hi @jrkirk61,

Great question! The short answer is no - I don't have access to a system running Torque myself, which makes it very difficult to test and develop. The long answer is maybe - I've had a chain of e-mails with someone from the University of Michigan who wanted to run it on Torque. We managed to get some way towards it working, but I haven't heard anything back from her since August so I'm not sure what the latest is at her end.

The way to set up Cluster Flow for this revolves around setting it to use GRIDEngine, but then using the @custom_job_submit_command parameter to customise it for Torque. On paper they sound fairly similar, so I had hopes that doing that might work (and if we could find the magical settings, creating a new default for Torque).

The latest command we had was the following:

echo "{{command}}" | qsub -V -A <proj> -l qos=<yourvar> -q <yourvar> -l procs={{cores}},pmem={{mem}} -d <directory?>

Then things became more complicated, as apparently Torque needs job id numbers to set up the dependencies. These aren't issued until after the job is submitted, so need to be collected at run time. This is what SLURM does, so totally possible, but we need to change the core and get it to parse the exact output given by Torque (the last communication we had I asked for a copy of this output).

Also, the job overview monitoring didn't work, but sounded like it should be possible again, by running the command qstat -x -f -1 -l -u <username> and parsing that in cluster flow. Again, a case of copying existing code for one of the other job managers and customising it until it works.

So - that's the current state of affairs. There's nothing to stop Cluster Flow from working with Torque, but as I can't test and develop on a Torque system myself it's a bit tricky to get it up and running. You're welcome to have a go at modifying the code and I'll do my best to help, but I can't guarantee that things will be finished any time in the near future.

Cheers,

Phil

[jrkirk61]

Phil,

Thank you for the information. At the moment, our lab needs to use something we can get up and running in the near future. However, we may come back to this at a later time and try to get it working on our Torque cluster.

Best, Jessica

[ewels]

Of course - no problem :) I'll leave this issue open for now in case anyone else is interested..

Phil

[ewels]

Closing this issue as inactive. If anyone else is interested feel free to reopen.