It's the memory, stupid! How to solve the "cannot change locale" problem.

[smoe]

I got STAR failing during indexing and the last information I got was about the problem top change the locale.

$ /gnu/store/zd7gdv173x41rmjdgc7qr3ycasna28hd-star-2.7.3a/bin/STAR --runThreadN 4 --genomeDir /home/moeller/FibrosisArrayExpress/output/star_index --readFilesIn /home/moeller/FibrosisArrayExpress/output/trimmed_reads/WT_Rep2_R.fastq.gz --readFilesCommand '/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin/gunzip  -c' --outSAMtype BAM Unsorted --outFileNamePrefix /home/moeller/FibrosisArrayExpress/output/mapped_reads/WT_Rep2_
sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/gnu/store/mmhimfwmmidf09jw1plw3aw1g1zn2nkh-bash-static-5.0.16/bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Feb 25 16:04:13 ..... started STAR run
Feb 25 16:04:13 ..... loading genome
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

The same job runs just fine when executed directly from the command line (copy'n'pasted). The warning was not fixable for me - but this is not what trigger the kill. It was the computer's memory limitation. Snakemake started the indexing process four times in parallel, even though the machine has only 64GB of RAM. The processes were hence killed by the operating system because it was running out of memory. With

execution:
    submit-to-cluster: no
    jobs: 1

everything ran just fine, coming from jobs: 4.

These are the rules

    rules:
      __default__:
         threads: 4
         memory: 16G
      star_index:
         threads: 4
         memory: 36G
#       salmon_index:
#         threads: 8
#         memory: 4G
#       salmon_quant:
#         threads: 8
#         memory: 5G
      star_map:
         threads: 1
         memory: 48G

Is there something I can do to chase this up? Having only 1 job at the time running is obviously not ideal.

Many thanks! Steffen

Sidenote: The repeated restarts (also me having to remove the snakemake's lock directory) have triggered this error:

$ tail -f /home/moeller/FibrosisArrayExpress/output/logs/samtools_sort_PR_Rep3.log
[E::hts_open_format] Failed to open file "/home/moeller/FibrosisArrayExpress/output/mapped_reads/PR_Rep3_Aligned.sortedByCoord.out.bam.tmp.0000.bam" : File exists
samtools sort: failed to create temporary file "/home/moeller/FibrosisArrayExpress/output/mapped_reads/PR_Rep3_Aligned.sortedByCoord.out.bam.tmp.0000.bam": File exists

Maybe these .bam files could be removed upfront?

[smoe]

Your snakemake magic is in pigx_rnaseq.py, right? I am not the ultimate snakemake person [have no idea about it] but this file does not describe a constraint on resources as in https://stackoverflow.com/questions/48542333/snakemake-memory-limiting which would explain my "start all that jobs allows" observation.

The only place that I see the memory information used is for the submission to a cluster via qsub.

[rekado]

This is correct. I'm not sure I remember correctly, but at the time that pigx development started snakemake had no way to specify resource limits or estimates.

[borauyar]

Unfortunately, STAR needs about 30GB of RAM for indexing mouse/human genomes. That is the trade-off for the speed that it brings during the alignment step. The only workaround to use more than one threads would be to move on to a larger memory node.

[smoe]

I just wished this memory-info neglect for local executions would have been stated somewhere. Investigating this took a bit of time - not because this was difficult to find, just because it was so unexpected for me and the jobs do not fail early - c'mon, a workflow system that starts processes even though it knows they run out of memory. But you likely just run qsub with everything ... or have bigger machines. Or both.

Now, it seems like snakemake can do this resource management and pigx-rnaseq already knows everything it needs to know. From the hip, i.e. from stackoverflow, it looks like "not too bad" to add this. Would you feel prepared to accept respective patches? I could also help with access to a machine for a first-hand experience. Next week is already taken, but then I should be up for it.

[smoe]

I found https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html?highlight=memory#standard-resources and will see what I can do.

[borauyar]

Resources are now defined within each rule in the snakefile see: https://github.com/BIMSBbioinfo/pigx_rnaseq/commit/3ac5261aca9049bbf42810907dfc8bd502ba0ebe

[smoe]

Nice, thanks!