search

Lioscro / alaska-parse

Automated and friendly RNA-seq analysis (migrated to Parse Server)
http://alaska.caltech.edu
1 stars 0 forks source link

Alaska memory bug #52

Closed raymond91125 closed 4 years ago

raymond91125 commented 4 years ago

To WormBase & Alaska,

We recently messaged concerning a couple of bugs we encountered on Alaska (selecting a comparison genome bug, inputting paired reads bug).

We have since run into another error which occurs from the first quality control step (please see code below) - "samtools sort: couldn't allocate memory for bam_mem”.

We encountered this bug with a previous sample, and after rerunning several times on different days, the full analysis completed with no errors last Friday.

We are now experiencing this bug with every uploaded file (incl. ones which have previously worked).

Any help would be much appreciated!

Thank you!

Alex

raymond91125 commented 4 years ago

starting qc for project sx1YYTlKcs

starting qc for sample sample_2_folder

starting qc on sample sample_2_folder

bowtie2 -x /alaska/data/organisms/caenorhabditis/elegans/WS268.PRJNA13758/bowtie/caenorhabditis_elegans_WS268.PRJNA13758 -U /alaska/data/projects/sx1YYTlKcs/reads/sample_2_folder/sample_2_read_1.fastq.gz -S /alaska/data/projects/sx1YYTlKcs/qc/sample_2_folder/sample_2_folder_alignments.sam -u 100000 --threads 3 --verbose

process started 2020-02-04 08:02:14

sample_2_folder: Applying preset: 'sensitive' using preset menu 'V0' sample_2_folder: Final policy string: 'SEED=0;SEEDLEN=22;DPS=15;ROUNDS=2;IVAL=S,1,1.15' sample_2_folder: Input bt2 file: "/alaska/data/organisms/caenorhabditis/elegans/WS268.PRJNA13758/bowtie/caenorhabditis_elegans_WS268.PRJNA13758" sample_2_folder: Query inputs (DNA, FASTQ): sample_2_folder: /alaska/data/projects/sx1YYTlKcs/reads/sample_2_folder/sample_2_read_1.fastq.gz sample_2_folder: Quality inputs: sample_2_folder: Output file: "/alaska/data/projects/sx1YYTlKcs/qc/sample_2_folder/sample_2_folder_alignments.sam" sample_2_folder: Local endianness: little sample_2_folder: Sanity checking: disabled sample_2_folder: Assertions: disabled sample_2_folder: Entered driver(): 08:02:14 sample_2_folder: Creating PatternSource: 08:02:14 sample_2_folder: Opening hit output file: 08:02:14 sample_2_folder: About to initialize fw Ebwt: 08:02:14 sample_2_folder: Trying /alaska/data/organisms/caenorhabditis/elegans/WS268.PRJNA13758/bowtie/caenorhabditis_elegans_WS268.PRJNA13758 sample_2_folder: About to open input files: 08:02:14 sample_2_folder: Opening "/alaska/data/organisms/caenorhabditis/elegans/WS268.PRJNA13758/bowtie/caenorhabditis_elegans_WS268.PRJNA13758.1.bt2" sample_2_folder: Opening "/alaska/data/organisms/caenorhabditis/elegans/WS268.PRJNA13758/bowtie/caenorhabditis_elegans_WS268.PRJNA13758.2.bt2" sample_2_folder: Finished opening input files: 08:02:14 sample_2_folder: Reading header: 08:02:14 sample_2_folder: Headers: sample_2_folder: len: 100286401 sample_2_folder: bwtLen: 100286402 sample_2_folder: sz: 25071601 sample_2_folder: bwtSz: 25071601 sample_2_folder: lineRate: 6 sample_2_folder: offRate: 4 sample_2_folder: offMask: 0xfffffff0 sample_2_folder: ftabChars: 10 sample_2_folder: eftabLen: 20 sample_2_folder: eftabSz: 80 sample_2_folder: ftabLen: 1048577 sample_2_folder: ftabSz: 4194308 sample_2_folder: offsLen: 6267901 sample_2_folder: offsSz: 25071604 sample_2_folder: lineSz: 64 sample_2_folder: sideSz: 64 sample_2_folder: sideBwtSz: 48 sample_2_folder: sideBwtLen: 192 sample_2_folder: numSides: 522326 sample_2_folder: numLines: 522326 sample_2_folder: ebwtTotLen: 33428864 sample_2_folder: ebwtTotSz: 33428864 sample_2_folder: color: 0 sample_2_folder: reverse: 0 sample_2_folder: Reading plen (7): 08:02:14 sample_2_folder: About to initialize rev Ebwt: 08:02:14 sample_2_folder: About to open input files: 08:02:14 sample_2_folder: Opening "/alaska/data/organisms/caenorhabditis/elegans/WS268.PRJNA13758/bowtie/caenorhabditis_elegans_WS268.PRJNA13758.rev.1.bt2" sample_2_folder: Opening "/alaska/data/organisms/caenorhabditis/elegans/WS268.PRJNA13758/bowtie/caenorhabditis_elegans_WS268.PRJNA13758.rev.2.bt2" sample_2_folder: Finished opening input files: 08:02:14 sample_2_folder: Reading header: 08:02:14 sample_2_folder: Headers: sample_2_folder: len: 100286401 sample_2_folder: bwtLen: 100286402 sample_2_folder: sz: 25071601 sample_2_folder: bwtSz: 25071601 sample_2_folder: lineRate: 6 sample_2_folder: offRate: 4 sample_2_folder: offMask: 0xfffffff0 sample_2_folder: ftabChars: 10 sample_2_folder: eftabLen: 20 sample_2_folder: eftabSz: 80 sample_2_folder: ftabLen: 1048577 sample_2_folder: ftabSz: 4194308 sample_2_folder: offsLen: 6267901 sample_2_folder: offsSz: 25071604 sample_2_folder: lineSz: 64 sample_2_folder: sideSz: 64 sample_2_folder: sideBwtSz: 48 sample_2_folder: sideBwtLen: 192 sample_2_folder: numSides: 522326 sample_2_folder: numLines: 522326 sample_2_folder: ebwtTotLen: 33428864 sample_2_folder: ebwtTotSz: 33428864 sample_2_folder: color: 0 sample_2_folder: reverse: 1 sample_2_folder: Reading plen (7): 08:02:14 sample_2_folder: Dispatching to search driver: 08:02:14 sample_2_folder: Read 7 reference strings from 7 records: 08:02:14 sample_2_folder: About to open input files: 08:02:14 sample_2_folder: Opening "/alaska/data/organisms/caenorhabditis/elegans/WS268.PRJNA13758/bowtie/caenorhabditis_elegans_WS268.PRJNA13758.1.bt2" sample_2_folder: Opening "/alaska/data/organisms/caenorhabditis/elegans/WS268.PRJNA13758/bowtie/caenorhabditis_elegans_WS268.PRJNA13758.2.bt2" sample_2_folder: Finished opening input files: 08:02:14 sample_2_folder: Reading header: 08:02:14 sample_2_folder: Reading plen (7): 08:02:14 sample_2_folder: Reading rstarts (21): 08:02:14 sample_2_folder: Reading ebwt (33428864): 08:02:14 sample_2_folder: Reading fchr (5) sample_2_folder: Reading ftab (1048577): 08:02:14 sample_2_folder: Reading eftab (20): 08:02:14 sample_2_folder: Reading offs (626790132-bit words): 08:02:14 sample_2_folder: Headers: sample_2_folder: len: 100286401 sample_2_folder: bwtLen: 100286402 sample_2_folder: sz: 25071601 sample_2_folder: bwtSz: 25071601 sample_2_folder: lineRate: 6 sample_2_folder: offRate: 4 sample_2_folder: offMask: 0xfffffff0 sample_2_folder: ftabChars: 10 sample_2_folder: eftabLen: 20 sample_2_folder: eftabSz: 80 sample_2_folder: ftabLen: 1048577 sample_2_folder: ftabSz: 4194308 sample_2_folder: offsLen: 6267901 sample_2_folder: offsSz: 25071604 sample_2_folder: lineSz: 64 sample_2_folder: sideSz: 64 sample_2_folder: sideBwtSz: 48 sample_2_folder: sideBwtLen: 192 sample_2_folder: numSides: 522326 sample_2_folder: numLines: 522326 sample_2_folder: ebwtTotLen: 33428864 sample_2_folder: ebwtTotSz: 33428864 sample_2_folder: color: 0 sample_2_folder: reverse: 0 sample_2_folder: Ebwt (memory): sample_2_folder: zOff: 58040422 sample_2_folder: zEbwtByteOff: 19346793 sample_2_folder: zEbwtBpOff: 2 sample_2_folder: nPat: 7 sample_2_folder: plen: non-NULL, [0] = 15072434 sample_2_folder: rstarts: non-NULL, [0] = 0 sample_2_folder: ebwt: non-NULL, [0] = sample_2_folder: fchr: non-NULL, [0] = 0 sample_2_folder: ftab: non-NULL, [0] = 0 sample_2_folder: eftab: non-NULL, [0] = 25262265 sample_2_folder: offs: non-NULL, [0] = 82984580 sample_2_folder: About to open input files: 08:02:14 sample_2_folder: Opening "/alaska/data/organisms/caenorhabditis/elegans/WS268.PRJNA13758/bowtie/caenorhabditis_elegans_WS268.PRJNA13758.rev.1.bt2" sample_2_folder: Finished opening input files: 08:02:15 sample_2_folder: Reading header: 08:02:15 sample_2_folder: Reading plen (7): 08:02:15 sample_2_folder: Reading rstarts (21): 08:02:15 sample_2_folder: Reading ebwt (33428864): 08:02:15 sample_2_folder: Reading fchr (5) sample_2_folder: Reading ftab (1048577): 08:02:15 sample_2_folder: Reading eftab (20): 08:02:15 sample_2_folder: Headers: sample_2_folder: len: 100286401 sample_2_folder: bwtLen: 100286402 sample_2_folder: sz: 25071601 sample_2_folder: bwtSz: 25071601 sample_2_folder: lineRate: 6 sample_2_folder: offRate: 4 sample_2_folder: offMask: 0xfffffff0 sample_2_folder: ftabChars: 10 sample_2_folder: eftabLen: 20 sample_2_folder: eftabSz: 80 sample_2_folder: ftabLen: 1048577 sample_2_folder: ftabSz: 4194308 sample_2_folder: offsLen: 6267901 sample_2_folder: offsSz: 25071604 sample_2_folder: lineSz: 64 sample_2_folder: sideSz: 64 sample_2_folder: sideBwtSz: 48 sample_2_folder: sideBwtLen: 192 sample_2_folder: numSides: 522326 sample_2_folder: numLines: 522326 sample_2_folder: ebwtTotLen: 33428864 sample_2_folder: ebwtTotSz: 33428864 sample_2_folder: color: 0 sample_2_folder: reverse: 1 sample_2_folder: Ebwt (memory): sample_2_folder: zOff: 30707246 sample_2_folder: zEbwtByteOff: 10235739 sample_2_folder: zEbwtBpOff: 2 sample_2_folder: nPat: 7 sample_2_folder: plen: non-NULL, [0] = 15072434 sample_2_folder: rstarts: NULL sample_2_folder: ebwt: non-NULL, [0] = sample_2_folder: fchr: non-NULL, [0] = 0 sample_2_folder: ftab: non-NULL, [0] = 0 sample_2_folder: eftab: non-NULL, [0] = 12239418 sample_2_folder: offs: NULL sample_2_folder: 100000 reads; of these: sample_2_folder: 100000 (100.00%) were unpaired; of these: sample_2_folder: 9273 (9.27%) aligned 0 times sample_2_folder: 80065 (80.06%) aligned exactly 1 time sample_2_folder: 10662 (10.66%) aligned >1 times sample_2_folder: 90.73% overall alignment rate

process finished 2020-02-04 08:02:17

samtools sort /alaska/data/projects/sx1YYTlKcs/qc/sample_2_folder/sample_2_folder_alignments.sam -o /alaska/data/projects/sx1YYTlKcs/qc/sample_2_folder/sample_2_folder_sorted.bam -@ 2 -m 2G

process started 2020-02-04 08:02:18

sample_2_folder: samtools sort: couldn't allocate memory for bam_mem

process finished 2020-02-04 08:02:18

Traceback (most recent call last): File "run_qc.py", line 291, in run_qc(project, code=code, nthreads=nthreads) File "run_qc.py", line 247, in run_qc qc(sample, nthreads=nthreads) File "run_qc.py", line 212, in qc sam_files = samtools_sort(sample, code=code, nthreads=nthreads) File "run_qc.py", line 168, in samtools_sort run_sys(args, prefix=sample.name) File "/alaska/scripts/utilities.py", line 118, in run_sys raise Exception('command terminated with non-zero return code!\n{}'.format(output)) Exception: command terminated with non-zero return code!

samtools sort /alaska/data/projects/sx1YYTlKcs/qc/sample_2_folder/sample_2_folder_alignments.sam -o /alaska/data/projects/sx1YYTlKcs/qc/sample_2_folder/sample_2_folder_sorted.bam -@ 2 -m 2G# process started 2020-02-04 08:02:18

samtools sort: couldn't allocate memory for bam_mem

process finished 2020-02-04 08:02:18

Sentry is attempting to send 0 pending error messages Waiting up to 2 seconds Press Ctrl-C to quit

raymond91125 commented 4 years ago

Double system memory to 16GB. Reboot. User reported problem solved.