SPEAQeasy: portable LIBD RNA-seq pipeline using Nextflow. Check http://research.libd.org/SPEAQeasy-example/ for an example on how to use this pipeline and analyze the resulting output files.
One specific request was AWS, but there are a huge number of other executors people could request in the future.
I think the real goal here is to find an elegant way to balance:
keeping the executor modular conceptually (executor-related configuration and execution details should be separate from those regarding hardware resources or unrelated details)
providing executor-specific details like this that provide sensible default configuration, thus reducing set-up difficulty for users
Ideally we'd have a way to, at runtime or install time, specify to SPEAQeasy an arbitrary executor supported by Nextflow, and have things mostly work out of the box. Then for the most popular and in-demand executors (say SLURM, SGE, JHPCE, and AWS with or without docker/singularity containers), we could spend additional effort maintaining executor-specific configuration (like this or this).
I imagine that such a system would work by having a sort of "template configuration" with core specifications supported by all executors (i.e. cpus, memory, and probably others), and at install time add executor-specific details that at minimum would be executor.name. This system would probably break backwards compatibility though.
One specific request was AWS, but there are a huge number of other executors people could request in the future.
I think the real goal here is to find an elegant way to balance:
Ideally we'd have a way to, at runtime or install time, specify to SPEAQeasy an arbitrary executor supported by Nextflow, and have things mostly work out of the box. Then for the most popular and in-demand executors (say SLURM, SGE, JHPCE, and AWS with or without docker/singularity containers), we could spend additional effort maintaining executor-specific configuration (like this or this).
I imagine that such a system would work by having a sort of "template configuration" with core specifications supported by all executors (i.e.
cpus,memory, and probably others), and at install time add executor-specific details that at minimum would beexecutor.name. This system would probably break backwards compatibility though.