search

epi2me-labs / wf-human-variation

Other
86 stars 41 forks source link

Explicit support for SLURM and other schedulers #176

Closed adbeggs closed 2 months ago

adbeggs commented 2 months ago

Is your feature related to a problem?

Although you can submit to a slurm cluster, the subjobs are not submitted using SLURM meaning that it is less efficient

Describe the solution you'd like

Explicit support of job schedulers in tasks, along the line of NF-Core

Describe alternatives you've considered

Doing it myself, but I don't have the time!

Additional context

No response

cjw85 commented 2 months ago

Slurm is supported through Nextflow's integrations with slurm. The precise config file used is intimately reliant on the end-user's HPC systems. Its not something we can support to include configuration files for all our users (and neither do we condone the idea of people submitting private cluster config profiles to a public respository).

adbeggs commented 2 months ago

Hi Chris

That’s kind of what Nextflow already support here:

https://github.com/nf-core/configs

Looking at your Nextflow config, I would have to rewrite your pipeline to explicitly support Slurm as it is not mentioned in the Nextflow.config and so doesn’t support it out of the box (unlike nf-core)

Happy to do this myself

A

From: Chris Wright @.> Sent: Tuesday, April 23, 2024 1:27 PM To: epi2me-labs/wf-human-variation @.> Cc: Andrew Beggs (Cancer and Genomic Sciences) @.>; Author @.> Subject: Re: [epi2me-labs/wf-human-variation] Explicit support for SLURM and other schedulers (Issue #176)

CAUTION: This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.

Slurm is supported through Nextflow's integrations with slurm. The precise config file used is intimately reliant on the end-user's HPC systems. Its not something we can support to include configuration files for all our users (and neither do we condone the idea of people submitting private cluster config profiles to a public respository).

— Reply to this email directly, view it on GitHubhttps://github.com/epi2me-labs/wf-human-variation/issues/176#issuecomment-2072167433, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC7KTDFEQTPX5B4RO4E6XNDY6ZHPZAVCNFSM6AAAAABGUXF4P2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZSGE3DONBTGM. You are receiving this because you authored the thread.Message ID: @.***>

cjw85 commented 2 months ago

We're very aware of https://github.com/nf-core/configs, this is what we find it hard to condone. Many IT professionals would consider information in that repository a security risl for their infrastructure.

You don't need to rewrite the pipeline. You need only to provide an additional config file containing the configuration required.