arq5x / lumpy-sv

lumpy: a general probabilistic framework for structural variant discovery
MIT License
307 stars 119 forks source link

lumpyexpress segmentation fault on a subset of samples #314

Closed sofieobr closed 4 years ago

sofieobr commented 4 years ago

Hi,

I have run lumpyexpress on 24 BWA-MEM mapped sequencing files, all aligned to the same genome. In 2 out of 24 of the samples, lumpy is giving a segmentation fault. The fault occurs at the same scaffold in both files, and lumpy gives a partial output that stops at that scaffold. In the other 22 samples, lumpy ran correctly with no errors. I tried to re-run lumpy, and the same segmentation fault occurs every time, at the same place. I processed all 24 files identically, and had plenty of storage and ram (I ran it with 93GB of ram at one point, and it still had the same error occur). I ran the following lumpyexpress command:

/home/huws/bluvs002/lumpy-sv/bin/lumpyexpress 
-B "CHOK1_SF_unique_sorted_dup_RG.bam" -S "CHOK1_SF_splitters.bam" 
-D "CHOK1_SF_discordants.bam" -k -v -T temp -o "CHOK1_SF_lumpy2.vcf"

Even though I used the -k option, all of the temporary files were deleted as soon as the segmentation fault occurred.

Here is the output of lumpy:

Sourcing executables from /home/huws/bluvs002/lumpy-sv/bin/lumpyexpress.config ...

Checking for required python modules (/home/huws/bluvs002/mypython2.7/bin/python2)...

    create temporary directory
Calculating insert distributions... 
Library read groups: 1
Library read length: 90
done
0
Running LUMPY... 

/panfs/roc/groups/13/huws/bluvs002/lumpy-sv//bin/lumpy  \
    -t temp/CHOK1_SF_lumpy2.vcf \
    -msw 4 \
    -tt 0 \
     \
     \
     -pe bam_file:CHOK1_SF_discordants.bam,histo_file:temp/CHOK1_SF_lumpy2.vcf.sample1.lib1.x4.histo,
mean:474.955262533,stdev:38.2016793044,read_length:90,min_non_overlap:90,discordant_z:5,back_distance:10,weight:1,id:CHOK1_SF,min_mapping_threshold:20,read_group:1 \
     -sr bam_file:CHOK1_SF_splitters.bam,back_distance:10,min_mapping_threshold:20,weight:1,id:CHOK1_SF,min_clip:20 \
    > CHOK1_SF_lumpy2.vcf

Here is the error output:

Removed 297 outliers with isize >= 550
470 0
NW_020822366.1  1000000
NW_020822368.1  1000000
NW_020822369.1  1000000
NW_020822370.1  1000000
NW_020822371.1  1000000
NW_020822372.1  1000000
NW_020822373.1  1000000
NW_020822374.1  1000000
NW_020822375.1  1000000
NW_020822376.1  1000000
NW_020822377.1  1000000
NW_020822379.1  1000000
NW_020822380.1  1000000
NW_020822381.1  1000000
NW_020822382.1  1000000
NW_020822383.1  1000000
NW_020822385.1  1000000
NW_020822387.1  1000000
NW_020822388.1  1000000
NW_020822390.1  1000000
NW_020822391.1  1000000
NW_020822393.1  1000000
NW_020822394.1  1000000
NW_020822395.1  1000000
NW_020822396.1  1000000
NW_020822398.1  1000000
NW_020822399.1  1000000
NW_020822400.1  1000000
NW_020822401.1  1000000
NW_020822402.1  1000000
NW_020822403.1  1000000
/home/huws/bluvs002/lumpy-sv/bin/lumpyexpress: line 548: 74238 Segmentation fault      
(core dumped) $LUMPY $PROB_CURVE -t ${TEMP_DIR}/${OUTBASE} -msw 
$MIN_SAMPLE_WEIGHT -tt $TRIM_THRES $LUMPY_DEPTH_STRING $EXCLUDE_BED_FMT 
$LUMPY_DISC_STRING $EXCLUDE_BED_FMT $LUMPY_SPL_STRING > $OUTPUT

Do you know what the problem may be?

ryanlayer commented 4 years ago

Are those all of the scaffolds?

On Thu, Sep 5, 2019 at 9:56 AM sofieobr notifications@github.com wrote:

Hi,

I have run lumpyexpress on 24 BWA-MEM mapped sequencing files, all aligned to the same genome. In 2 out of 24 of the samples, lumpy is giving a segmentation fault. The fault occurs at the same scaffold in both files, and lumpy gives a partial output that stops at that scaffold. In the other 22 samples, lumpy ran correctly with no errors. I tried to re-run lumpy, and the same segmentation fault occurs every time, at the same place. I processed all 24 files identically, and had plenty of storage and ram (I ran it with 93GB of ram at one point, and it still had the same error occur). I ran the following lumpyexpress command:

/home/huws/bluvs002/lumpy-sv/bin/lumpyexpress -B "CHOK1_SF_unique_sorted_dup_RG.bam" -S "CHOK1_SF_splitters.bam" -D "CHOK1_SF_discordants.bam" -k -v -T temp -o "CHOK1_SF_lumpy2.vcf"

Even though I used the -k option, all of the temporary files were deleted as soon as the segmentation fault occurred.

Here is the output of lumpy:

Sourcing executables from /home/huws/bluvs002/lumpy-sv/bin/lumpyexpress.config ...

Checking for required python modules (/home/huws/bluvs002/mypython2.7/bin/python2)...

create temporary directory

Calculating insert distributions... Library read groups: 1 Library read length: 90 done 0 Running LUMPY...

/panfs/roc/groups/13/huws/bluvs002/lumpy-sv//bin/lumpy \ -t temp/CHOK1_SF_lumpy2.vcf \ -msw 4 \ -tt 0 \ \ \ -pe bam_file:CHOK1_SF_discordants.bam,histo_file:temp/CHOK1_SF_lumpy2.vcf.sample1.lib1.x4.histo, mean:474.955262533,stdev:38.2016793044,read_length:90,min_non_overlap:90,discordant_z:5,back_distance:10,weight:1,id:CHOK1_SF,min_mapping_threshold:20,read_group:1 \ -sr bam_file:CHOK1_SF_splitters.bam,back_distance:10,min_mapping_threshold:20,weight:1,id:CHOK1_SF,min_clip:20 \

CHOK1_SF_lumpy2.vcf

Here is the error output:

Removed 297 outliers with isize >= 550 470 0 NW_020822366.1 1000000 NW_020822368.1 1000000 NW_020822369.1 1000000 NW_020822370.1 1000000 NW_020822371.1 1000000 NW_020822372.1 1000000 NW_020822373.1 1000000 NW_020822374.1 1000000 NW_020822375.1 1000000 NW_020822376.1 1000000 NW_020822377.1 1000000 NW_020822379.1 1000000 NW_020822380.1 1000000 NW_020822381.1 1000000 NW_020822382.1 1000000 NW_020822383.1 1000000 NW_020822385.1 1000000 NW_020822387.1 1000000 NW_020822388.1 1000000 NW_020822390.1 1000000 NW_020822391.1 1000000 NW_020822393.1 1000000 NW_020822394.1 1000000 NW_020822395.1 1000000 NW_020822396.1 1000000 NW_020822398.1 1000000 NW_020822399.1 1000000 NW_020822400.1 1000000 NW_020822401.1 1000000 NW_020822402.1 1000000 NW_020822403.1 1000000 /home/huws/bluvs002/lumpy-sv/bin/lumpyexpress: line 548: 74238 Segmentation fault (core dumped) $LUMPY $PROB_CURVE -t ${TEMP_DIR}/${OUTBASE} -msw $MIN_SAMPLE_WEIGHT -tt $TRIM_THRES $LUMPY_DEPTH_STRING $EXCLUDE_BED_FMT $LUMPY_DISC_STRING $EXCLUDE_BED_FMT $LUMPY_SPL_STRING > $OUTPUT

Do you know what the problem may be?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/arq5x/lumpy-sv/issues/314?email_source=notifications&email_token=AAEUGUOLWCG5EBKRHHXBVP3QIET3ZA5CNFSM4IT7T32KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HJSSUOQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEUGUJAGYUKTTQC5IMSEN3QIET3ZANCNFSM4IT7T32A .

sofieobr commented 4 years ago

No, the error occurs after the first 31 scaffolds. The genome has 1,830 scaffolds in total.

ryanlayer commented 4 years ago

That segfault is almost always because of memory issues, but the memory issue usually comes up after all of the scaffolds have been processed. Is there any way you can watch the memory usage?

On Thu, Sep 5, 2019 at 3:48 PM sofieobr notifications@github.com wrote:

No, the error occurs after the first 31 scaffolds. The genome has 1,830 scaffolds in total.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/arq5x/lumpy-sv/issues/314?email_source=notifications&email_token=AAEUGUJMJQ5E3WA3FXT7JUDQIF5BBA5CNFSM4IT7T32KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6A5OCY#issuecomment-528602891, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEUGUJGUHFZWAUEONLNLZTQIF5BBANCNFSM4IT7T32A .

sofieobr commented 4 years ago

I submit lumpy as a job on a supercomputer with node sharing. I am not sure if I can track memory usage over time, but it does tell me the memory used once the job quits from the segfault. Lumpyexpress used 0.629 GB of memory out of the 21GB I requested. I re-ran it twice more with 97 GB and 130 GB of memory and it quit from the segfault both times after using the same amount (0.629GB).

ryanlayer commented 4 years ago

Are these all Illumina paired-end libraries?

On Sep 5, 2019, at 4:16 PM, sofieobr notifications@github.com wrote:

I submit lumpy as a job on a supercomputer with node sharing. I am not sure if I can track memory usage over time, but it does tell me the memory used once the job quits from the segfault. Lumpyexpress used 0.629 GB of memory out of the 21GB I requested. I re-ran it twice more with 97 GB and 130 GB of memory and it quit from the segfault both times after using the same amount (0.629GB).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

sofieobr commented 4 years ago

Yes, they are all Illumina paired-end libraries. In addition, one of the files that failed was part of a set of 6 that was library prepped and sequenced the same way. Lumpy ran without issue on the other 5 files.

sofieobr commented 4 years ago

I took a look at the core files from the segmentation fault for the 2 samples that failed. Here are the issues for each sample:

Sample 1:

#0  0x000000000041eb12 in SV_BreakPoint::trim_interval (interval_v=interval_v@entry=0x28e3f80, size=size@entry=126, trim_start=trim_start@entry=0x7ffe8b580bb0, trim_end=trim_end@entry=0x7ffe8b580bb4) at SV_BreakPoint.cpp:695
#1  0x000000000041fe39 in SV_BreakPoint::trim_intervals (this=this@entry=0x26703e0) at SV_BreakPoint.cpp:624
#2  0x000000000040821d in main (argc=<optimized out>, argv=<optimized out>) at lumpy.cpp:598

Sample 2:

#0  0x000000000041eb12 in SV_BreakPoint::trim_interval (interval_v=interval_v@entry=0x63a7da0, size=size@entry=167, trim_start=trim_start@entry=0x7ffdc4129d40, trim_end=trim_end@entry=0x7ffdc4129d44) at SV_BreakPoint.cpp:695
#1  0x000000000041fe39 in SV_BreakPoint::trim_intervals (this=this@entry=0x5e02940) at SV_BreakPoint.cpp:624
#2  0x000000000040821d in main (argc=<optimized out>, argv=<optimized out>) at lumpy.cpp:598

Is there a way to determine a problematic alignment from this information?

ryanlayer commented 4 years ago

That’s a good idea. Can you make a bam with just the scaffold that failed and run lumpy in that? If it still fails you can send me the bam and I can run down the error.

On Sep 9, 2019, at 11:27 AM, sofieobr notifications@github.com wrote:

I took a look at the core files from the segmentation fault for the 2 samples that failed. Here are the issues for each sample:

Sample 1:

0 0x000000000041eb12 in SV_BreakPoint::trim_interval (interval_v=interval_v@entry=0x28e3f80, size=size@entry=126, trim_start=trim_start@entry=0x7ffe8b580bb0, trim_end=trim_end@entry=0x7ffe8b580bb4) at SV_BreakPoint.cpp:695

1 0x000000000041fe39 in SV_BreakPoint::trim_intervals (this=this@entry=0x26703e0) at SV_BreakPoint.cpp:624

2 0x000000000040821d in main (argc=, argv=) at lumpy.cpp:598

Sample 2:

0 0x000000000041eb12 in SV_BreakPoint::trim_interval (interval_v=interval_v@entry=0x63a7da0, size=size@entry=167, trim_start=trim_start@entry=0x7ffdc4129d40, trim_end=trim_end@entry=0x7ffdc4129d44) at SV_BreakPoint.cpp:695

1 0x000000000041fe39 in SV_BreakPoint::trim_intervals (this=this@entry=0x5e02940) at SV_BreakPoint.cpp:624

2 0x000000000040821d in main (argc=, argv=) at lumpy.cpp:598

Is there a way to determine a problematic alignment from this information?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

sofieobr commented 4 years ago

I re-ran lumpy for just that scaffold in sample 1 and it failed in the same way. How can I send you the bam? It is 0.24GB. Also, here is the core dump information from the last run if that is helpful:

#0  0x000000000041eb12 in SV_BreakPoint::trim_interval (interval_v=interval_v@entry=0x17e0e00, size=size@entry=124, trim_start=trim_start@entry=0x7ffdabbfe6e0, trim_end=trim_end@entry=0x7ffdabbfe6e4) at SV_BreakPoint.cpp:695
#1  0x000000000041fe39 in SV_BreakPoint::trim_intervals (this=this@entry=0x165cb90) at SV_BreakPoint.cpp:624
#2  0x000000000040821d in main (argc=<optimized out>, argv=<optimized out>) at lumpy.cpp:598
ryanlayer commented 4 years ago

Please send to ryan layer at gmail.

On Sep 9, 2019, at 12:28 PM, sofieobr notifications@github.com wrote:

I re-ran lumpy for just that scaffold in sample 1 and it failed in the same way. How can I send you the bam? It is 0.24GB. Also, here is the core dump information from the last run if that is helpful:

0 0x000000000041eb12 in SV_BreakPoint::trim_interval (interval_v=interval_v@entry=0x17e0e00, size=size@entry=124, trim_start=trim_start@entry=0x7ffdabbfe6e0, trim_end=trim_end@entry=0x7ffdabbfe6e4) at SV_BreakPoint.cpp:695

1 0x000000000041fe39 in SV_BreakPoint::trim_intervals (this=this@entry=0x165cb90) at SV_BreakPoint.cpp:624

2 0x000000000040821d in main (argc=, argv=) at lumpy.cpp:598

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

sofieobr commented 4 years ago

I fixed the problem by re-downloading and re-compiling lumpy. I'm still not sure why the previous version worked for 22/24 samples and errored in the same place for the other two, but the issue was resolved. Thank you for your help!

linfeng0204 commented 3 years ago

I fixed the problem by re-downloading and re-compiling lumpy. I'm still not sure why the previous version worked for 22/24 samples and errored in the same place for the other two, but the issue was resolved. Thank you for your help!

Thanks for the info! I encountered the same problem with lumpy 0.2.13 and resolved the problem with lumpy 0.3.0.

ryanlayer commented 3 years ago

Great! Thanks.

On Mon, Nov 2, 2020 at 5:38 PM linfeng0204 notifications@github.com wrote:

I fixed the problem by re-downloading and re-compiling lumpy. I'm still not sure why the previous version worked for 22/24 samples and errored in the same place for the other two, but the issue was resolved. Thank you for your help!

Thanks for the info! I encountered the same problem with lumpy 0.2.13 and resolved the problem with lumpy 0.3.0.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/arq5x/lumpy-sv/issues/314#issuecomment-720806693, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEUGUIOEAJNLC55MJJLJN3SN5GIFANCNFSM4IT7T32A .