search

brian-cleary / LatentStrainAnalysis

Partitioning and analysis methods for large, complex sequence datasets
MIT License
37 stars 20 forks source link

Torque Scheduler Support #4

Open mbcouger opened 8 years ago

mbcouger commented 8 years ago

Hello LatentStrain Team,

Currently our HPC uses the Torque scheduling software for job execution, as do many other HPC systems. Is there currently any support for this architecture. If not would you be able to show me the individual commands executed to perform the "Analyzing Data sets" so I can make my own submission scripts.

Many Thanks, Brian

brian-cleary commented 8 years ago

Hi Brian,

Sorry for the earlier non-reply. That's great to hear that you're doing it!

If you haven't already done so, you can see how the various job submission scripts are created in LSF_scripts/create_jobs.py.

You'll be able to create some of these submission scripts without having a project already going, and for others you'll need certain directory structures and files in place (the size of job arrays is determined from file counts in many cases). One thing you could try is to setup and run the test data (no distributed computing required), then, with all these directories and files in place, run the various commands to create the submission scripts. This way you'll get to see what the final submission script at each step is supposed to look like (keeping in mind that the size of the array should be variable).

I hope this helps!

On Wed, Oct 28, 2015 at 6:14 PM, mbcouger notifications@github.com wrote:

Hello Latent Strain,

Working on doing this if it has not been done.

Cheers Brian

— Reply to this email directly or view it on GitHub https://github.com/brian-cleary/LatentStrainAnalysis/issues/4#issuecomment-152014147 .

mbcouger commented 8 years ago

Hello Brian,

No Issue at all, really glad to see this software released an the Nature Biotech Publication, scaling has been a huge problem for quiet some time with metagnomemics. I have started looking through the python scripts and how to adapt them to a different scheduler. Once I am further along or have any questions I will send you an update.

Cheers, Brian

agofton commented 5 years ago

Hello all,

FYI I am currently working on adapting the LSF submission script to SLURM for my cluster. At this point not sure whether I will reconfigure create_job.py or just rewrite the individual job scripts calling SLURM variables rather than BSUB. Either way I will keep this thread posted on any progress or issues.

Cheers,