search

BIMSBbioinfo / pigx_rnaseq

Bulk RNA-seq Data Processing, Quality Control, and Downstream Analysis Pipeline
GNU General Public License v3.0
20 stars 11 forks source link

all rules should have `resources` #103

Closed rekado closed 2 years ago

rekado commented 2 years ago

On a system with limited resources snakemake will sometimes schedule too many concurrent tasks, which leads to unexpected memory exhaustion. This is hard to reproduce, but it is a result of not having a resources section for all rules.

We could avoid this by adding a resources section to all rules, not just the known memory intensive rules.

smoe commented 2 years ago

You observed that behaviour for local executions, right? With a regular 64GByte memory machine the default settings will start as many STAR instances as there are number of jobs allowed, which was 6 for as. This happily violated the constraint that 6*16GB (memory for star_map) >64GB (memory of machine) contraint. Something is wrong somewhere IMHO. What helped me to just go somewhere is to limit the number of jobs to 2. We were then a bit suprised to find four bamCoverage jobs, but these don't nee much memory anyway, but should not be allowed since these are four jobs now running in parallel. Is there something I shall test to help locating the issue?

rekado commented 2 years ago

We've been using hisat2 for our tests on memory-restricted machines. (STAR has somewhat adventurous memory requirements, which makes it hard to use it in our current round of tests.)

Something is wrong somewhere IMHO

Yeah, I agree :)

I would like to avoid having to globally limit the number of jobs, because the scheduler should be able to respect our constraints. That's why I'd like us to mark up all rules first, so that we can rule out the bug in this obvious place.

smoe commented 2 years ago

I have updated snakemake to version 6.9.1 on Debian and do not see this problem any more. May I ask about what platform you made your observations with?

rekado commented 2 years ago

We had these problems on AWS. The problem is not easily reproduced because it depends on when snakemake schedules tasks on the actual machine.

I'll upgrade snakemake as well now.