search

STOmics / SAW

GNU General Public License v3.0
136 stars 34 forks source link

Question in mapping: [malloc(): memory corruption] #107

Open yzhirui opened 4 months ago

yzhirui commented 4 months ago

Hi, I encountered an error when do the mapping with SAW, which report the malloc(): memory corruption, . The memory value gived follows CIDcount result. And I also tried v06, it cannnot be solved. Could you help me to check out the possible reason? Thank you a lot!

The version of SAW is: 07.1.1 (the lastest one). My data is single-end.

The cmd is：

/opt/saw_st_software/pipeline/mapping/bcSTAR --outSAMattributes spatial \
    --outSAMtype BAM SortedByCoordinate \
    --genomeDir /data/user/Reference/Stereo_seq_mouse_mm10/STAR_SJ100 \
    --runThreadN 8 \
    --outFileNamePrefix /data/user/StereoSeq_Data/02.mapping/01. \
    --sysShell /bin/bash --stParaFile /data/user/StereoSeq_Data/02.mapping/01.bcPara \
    --readNameSeparator " " \
    --limitBAMsortRAM 38582880124 \
    --limitOutSJcollapsed 10000000 \
    --limitIObufferSize=280000000 \
    --outBAMsortingBinsN 50

The 01.bcPara is:

in=/data/yangzhirui/StereoSeq_Data/00.splitmask/01.B02605E3.barcodeToPos.bin
in1=/data/yangzhirui/StereoSeq_Data/00.FQ/FQ_01.list
out=B02605E3.bc.out01
barcodeReadsCount=/data/user/StereoSeq_Data/02.mapping/01.barcodeReadsCount.txt
action=4
barcodeStart=0
barcodeLen=24
umiStart=25
umiLen=10
umiRead=1
mismatch=5
bcNum=623569440
polyAnum=15
mismatchInPolyA=2
rRNAremove

The log info is: --- compile: Fri Mar 15 17:00:33 CST 2024 --- Info: v2.1.1 (based on STAR 2.7.2b) is released after optimization by BGI. - May 13 16:23:06 ..... started bcSTAR run --- WARN: parameter - out is not supported in this version. --- WARN: parameter - action is not supported in this version. --- WARN: parameter - umiRead is not supported in this version. --- Info: read-2 /data/user/StereoSeq_Data/00.FQ/fastq/barcode_81/sample_L01_81_1.fq.gz --- Info: read-2 /data/user/StereoSeq_Data/00.FQ/fastq/barcode_82/sample_L01_82_1.fq.gz --- Info: read-2 /data/user/StereoSeq_Data/00.FQ/fastq/barcode_83/sample_L01_83_1.fq.gz --- Info: read-2 /data/user/StereoSeq_Data/00.FQ/fastq/barcode_84/sample_L01_84_1.fq.gz --- Info: read-2 /data/user/StereoSeq_Data/00.FQ/fastq/barcode_85/sample_L01_85_1.fq.gz --- Info: read-2 /data/user/StereoSeq_Data/00.FQ/fastq/barcode_86/sample_L01_86_1.fq.gz --- Info: read-2 /data/user/StereoSeq_Data/00.FQ/fastq/barcode_87/sample_L01_87_1.fq.gz --- Info: read-2 /data/user/StereoSeq_Data/00.FQ/fastq/barcode_88/sample_L01_88_1.fq.gz --- Info: time for creating barcode map - 28 s, size of bcmap - 37002203 Error in `/opt/saw_st_software/pipeline/mapping/bcSTAR': malloc(): memory corruption: 0x0000000002325780

======= Backtrace: ========= /opt/lib/libc.so.6(+0x82b36)[0x147a23208b36] /opt/lib/libc.so.6(libc_malloc+0x4c)[0x147a2320b78c] /opt/lib/libstdc++.so.6(_Znwm+0x15)[0x147a23f607e5] /opt/saw_st_software/pipeline/mapping/bcSTAR[0x49889f] /opt/saw_st_software/pipeline/mapping/bcSTAR[0x45a801] /opt/saw_st_software/pipeline/mapping/bcSTAR[0x41d710] /opt/lib/libc.so.6(libc_start_main+0xf5)[0x147a231a8555] /opt/saw_st_software/pipeline/mapping/bcSTAR[0x421c7f] ======= Memory map: ======== 00400000-00683000 r-xp 00000000 07:00 15363 /opt/saw_st_software/pipeline/mapping/bcSTAR 00882000-00883000 r--p 00282000 07:00 15363 /opt/saw_st_software/pipeline/mapping/bcSTAR 00883000-0088e000 rw-p 00283000 07:00 15363 /opt/saw_st_software/pipeline/mapping/bcSTAR 0088e000-00a90000 rw-p 00000000 00:00 0 02308000-0237c000 rw-p 00000000 00:00 0 [heap] 1475b8000000-1475b8021000 rw-p 00000000 00:00 0 1475b8021000-1475bc000000 ---p 00000000 00:00 0 1475bffff000-147600000000 rw-p 00000000 00:00 0 147600000000-14760006b000 rw-p 00000000 00:00 0 14760006b000-147604000000 ---p 00000000 00:00 0 147605363000-147605364000 ---p 00000000 00:00 0 147605364000-147a07d75000 rw-p 00000000 00:00 0 147a07d75000-147a07d76000 ---p 00000000 00:00 0 147a07d76000-147a23186000 rw-p 00000000 00:00 0 147a23186000-147a2334a000 r-xp 00000000 07:00 11929 /opt/lib/libc.so.6 147a2334a000-147a23549000 ---p 001c4000 07:00 11929 /opt/lib/libc.so.6 147a23549000-147a2354d000 r--p 001c3000 07:00 11929 /opt/lib/libc.so.6 147a2354d000-147a2354f000 rw-p 001c7000 07:00 11929 /opt/lib/libc.so.6 147a2354f000-147a23554000 rw-p 00000000 00:00 0 147a23554000-147a2356b000 r-xp 00000000 07:00 12076 /opt/lib/libpthread.so.0 147a2356b000-147a2376a000 ---p 00017000 07:00 12076 /opt/lib/libpthread.so.0 147a2376a000-147a2376b000 r--p 00016000 07:00 12076 /opt/lib/libpthread.so.0 147a2376b000-147a2376c000 rw-p 00017000 07:00 12076 /opt/lib/libpthread.so.0 147a2376c000-147a23770000 rw-p 00000000 00:00 0 147a23770000-147a23786000 r-xp 00000000 07:00 11950 /opt/lib/libgcc_s.so.1 147a23786000-147a23985000 ---p 00016000 07:00 11950 /opt/lib/libgcc_s.so.1 147a23985000-147a23986000 r--p 00015000 07:00 11950 /opt/lib/libgcc_s.so.1 147a23986000-147a23987000 rw-p 00016000 07:00 11950 /opt/lib/libgcc_s.so.1 147a23987000-147a239b8000 r-xp 00000000 07:00 11952 /opt/lib/libgomp.so.1 147a239b8000-147a23bb8000 ---p 00031000 07:00 11952 /opt/lib/libgomp.so.1 147a23bb8000-147a23bb9000 r--p 00031000 07:00 11952 /opt/lib/libgomp.so.1 147a23bb9000-147a23bba000 rw-p 00032000 07:00 11952 /opt/lib/libgomp.so.1 147a23bba000-147a23cbb000 r-xp 00000000 07:00 11980 /opt/lib/libm.so.6 147a23cbb000-147a23eba000 ---p 00101000 07:00 11980 /opt/lib/libm.so.6 147a23eba000-147a23ebb000 r--p 00100000 07:00 11980 /opt/lib/libm.so.6 147a23ebb000-147a23ebc000 rw-p 00101000 07:00 11980 /opt/lib/libm.so.6 147a23ebc000-147a24085000 r-xp 00000000 07:00 12085 /opt/lib/libstdc++.so.6 147a24085000-147a24284000 ---p 001c9000 07:00 12085 /opt/lib/libstdc++.so.6 147a24284000-147a2428f000 r--p 001c8000 07:00 12085 /opt/lib/libstdc++.so.6 147a2428f000-147a24292000 rw-p 001d3000 07:00 12085 /opt/lib/libstdc++.so.6 147a24292000-147a24295000 rw-p 00000000 00:00 0 147a24295000-147a242a1000 r-xp 00000000 07:00 11943 /opt/lib/libdeflate.so.0 147a242a1000-147a244a0000 ---p 0000c000 07:00 11943 /opt/lib/libdeflate.so.0 147a244a0000-147a244a1000 r--p 0000b000 07:00 11943 /opt/lib/libdeflate.so.0 147a244a1000-147a244a2000 rw-p 0000c000 07:00 11943 /opt/lib/libdeflate.so.0 147a244a2000-147a248c5000 r-xp 00000000 07:00 11964 /opt/lib/libhdf5.so.200 147a248c5000-147a24ac4000 ---p 00423000 07:00 11964 /opt/lib/libhdf5.so.200 147a24ac4000-147a24ace000 r--p 00422000 07:00 11964 /opt/lib/libhdf5.so.200 147a24ace000-147a24ad7000 rw-p 0042c000 07:00 11964 /opt/lib/libhdf5.so.200 147a24ad7000-147a24ada000 rw-p 00000000 00:00 0 147a24ada000-147a24aea000 r-xp 00000000 07:00 11945 /opt/lib/libdouble-conversion.so.1 147a24aea000-147a24ce9000 ---p 00010000 07:00 11945 /opt/lib/libdouble-conversion.so.1 147a24ce9000-147a24cea000 r--p 0000f000 07:00 11945 /opt/lib/libdouble-conversion.so.1 147a24cea000-147a24ceb000 rw-p 00010000 07:00 11945 /opt/lib/libdouble-conversion.so.1 147a24ceb000-147a24d00000 r-xp 00000000 07:00 12105 /opt/lib/libz.so.1 147a24d00000-147a24eff000 ---p 00015000 07:00 12105 /opt/lib/libz.so.1 147a24eff000-147a24f00000 r--p 00014000 07:00 12105 /opt/lib/libz.so.1 147a24f00000-147a24f01000 rw-p 00015000 07:00 12105 /opt/lib/libz.so.1 147a24f01000-147a24f03000 r-xp 00000000 07:00 11944 /opt/lib/libdl.so.2 147a24f03000-147a25103000 ---p 00002000 07:00 11944 /opt/lib/libdl.so.2 147a25103000-147a25104000 r--p 00002000 07:00 11944 /opt/lib/libdl.so.2 147a25104000-147a25105000 rw-p 00003000 07:00 11944 /opt/lib/libdl.so.2 147a25105000-147a25127000 r-xp 00000000 07:00 234590 /usr/lib64/ld-2.17.so 147a2531a000-147a25326000 rw-p 00000000 00:00 0 147a25326000-147a25327000 r--p 00021000 07:00 234590 /usr/lib64/ld-2.17.so 147a25327000-147a25328000 rw-p 00022000 07:00 234590 /usr/lib64/ld-2.17.so 147a25328000-147a25329000 rw-p 00000000 00:00 0 7fffb81b6000-7fffb81d7000 rw-p 00000000 00:00 0 [stack] 7fffb81f8000-7fffb81fb000 r--p 00000000 00:00 0 [vvar] 7fffb81fb000-7fffb81fd000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] /usr/local/bin/mapping: line 1: 1094515 Aborted /opt/saw_st_software/pipeline/mapping/bcSTAR $*

TheSallyGardens commented 4 months ago

@yzhirui Hi. Checking the error log may indicate insufficient allocated memory.The limitBAMsortRAM parameter can be set to 63168332971 according to the tutorial.I hope it can help you.

yzhirui commented 4 months ago

@TheSallyGardens Hi, thanks for your kind reply!

I changed the limitBAMsortRAM parameter as you considered, the script now is:

 mapping --runMode alignReads\
 --outSAMattributes spatial \
 --outSAMtype BAM SortedByCoordinate \
 --genomeDir ${reference_path} \
 --runThreadN 8 \
 --outFileNamePrefix ${work_path}/01.mapping/${idx}. \
 --sysShell /bin/bash \
 --stParaFile ${work_path}/01.mapping/${idx}.test.bcPara \
 --readNameSeparator \" \" \
 --limitBAMsortRAM 63168332971 \
 --limitOutSJcollapsed 10000000 \
 --limitIObufferSize=280000000 \
 --outBAMsortingBinsN 50 \
> ${work_path}/01.mapping/${idx}_barcodeMap.stat

However, it didn't work. The reported error is still :

*** Error in `/opt/saw_st_software/pipeline/mapping/bcSTAR': malloc(): memory corruption: 0x00000000023e4770 ***\
======= Backtrace: =========
/opt/lib/libc.so.6(+0x82b36)[0x14b40fe9fb36]
/opt/lib/libc.so.6(__libc_malloc+0x4c)[0x14b40fea278c]
/opt/lib/libstdc++.so.6(_Znwm+0x15)[0x14b410bf77e5]
/opt/saw_st_software/pipeline/mapping/bcSTAR[0x49889f]
/opt/saw_st_software/pipeline/mapping/bcSTAR[0x45a801]
/opt/saw_st_software/pipeline/mapping/bcSTAR[0x41d710]
/opt/lib/libc.so.6(__libc_start_main+0xf5)[0x14b40fe3f555]
/opt/saw_st_software/pipeline/mapping/bcSTAR[0x421c7f]
======= Memory map: ========
00400000-00683000 r-xp 00000000 fd:00 605210                             /opt/saw_st_software/pipeline/mapping/bcSTAR
00882000-00883000 r--p 00282000 fd:00 605210                             /opt/saw_st_software/pipeline/mapping/bcSTAR
00883000-0088e000 rw-p 00283000 fd:00 605210                             /opt/saw_st_software/pipeline/mapping/bcSTAR
0088e000-00a90000 rw-p 00000000 00:00 0 
023c1000-02435000 rw-p 00000000 00:00 0                                  [heap]
......
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
/usr/local/bin/mapping: line 1: 591945 Aborted                 (core dumped) /opt/saw_st_software/pipeline/mapping/bcSTAR $*

Same error as before. I offered 450G memory and 8 cpu cores, respectively. Do you have any other suggestions about the possible reason?

Looking forward to your reply. Yours.

TheSallyGardens commented 4 months ago

@yzhirui Hi! Could you set mismatch to 1? I suspect this is causing the high memory requirements.