search

ijuric / MAPS

18 stars 11 forks source link

pysam.sort issue #1

Open sethb3744 opened 5 years ago

sethb3744 commented 5 years ago

Hi,

I ran the MAPS pipeline attached and it fails with the log below. I already tried troubleshooting the error by shortening the IJ052.R1.fastq.gz file from hundreds of millions of reads to one hundred thousand reads and then to one million reads. Both of the last two worked perfectly, but the original large file produced the following error related to the pysam.sort function:

[sethberg@iq218 bin]$ bash run_pipeline_IJ052.sh Tue Nov 20 09:32:44 2018 starting mapping and filtering operation Tue Nov 20 09:32:44 2018 calling bwa for /shen/shenlabstore2/sethberg/projects/hHumanFetal/fastp_output/IJ052.fastp.100bp.R1.fastq.gz Tue Nov 20 14:04:01 2018 calling bwa for /shen/shenlabstore2/sethberg/projects/hHumanFetal/fastp_output/IJ052.fastp.100bp.R2.fastq.gz Tue Nov 20 15:26:08 2018 calling samtools sort for /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/tempfiles/IJ052.fastp.100bp.R1.fastq.gz.bwa.sam storing in /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/tempfiles/IJ052.fastp.100bp.R1.fastq.gz.bwa.sam.srtn Tue Nov 20 15:45:09 2018 calling samtools sort for /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/tempfiles/IJ052.fastp.100bp.R2.fastq.gz.bwa.sam storing in /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/tempfiles/IJ052.fastp.100bp.R2.fastq.gz.bwa.sam.srtn [W::sam_read1] Parse error at line 13480925 Traceback (most recent call last): File "/netapp/home/sethberg/dependencies/MAPS/bin/feather/feather_pipe", line 116, in main() File "/netapp/home/sethberg/dependencies/MAPS/bin/feather/feather_pipe", line 49, in main filter_output_filename = filter_main(fastq1, fastq2, bwa_index, mapq, outdir, prefix, threads, to_file = False) File "/netapp/home/sethberg/dependencies/MAPS/bin/feather/feather_filter_chr.py", line 53, in filter_main pysam.sort("-o", bwa_sorted_filename , "-n", "-@", str(threads), bwa_filename) File "/netapp/home/sethberg/dependencies/Python-3.4.9/install_madebyseth/lib/python3.4/site-packages/pysam/utils.py", line 75, in call stderr)) pysam.utils.SamtoolsError: 'samtools returned with error 1: stdout=, stderr=samtools sort: truncated file. Aborting\n' sed: can't read /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/IJ052.feather.qc: No such file or directory

ijuric commented 5 years ago

Hi Seth, I’m not near my computer until Sunday (thanksgiving) so I can’t reproduce this error right now. I’m ccing Armen to see if he might have some insight into this. He ran maps on ij052 set on our side.

On Tue, Nov 20, 2018 at 7:05 PM sethb3744 notifications@github.com wrote:

Hi,

I ran the MAPS pipeline attached and it fails with the log below. I already tried troubleshooting the error by shortening the IJ052.R1.fastq.gz file from hundreds of millions of reads to one hundred thousand reads and then to one million reads. Both of the last two worked perfectly, but the original large file produced the following error related to the pysam.sort function:

[sethberg@iq218 bin]$ bash run_pipeline_IJ052.sh Tue Nov 20 09:32:44 2018 starting mapping and filtering operation Tue Nov 20 09:32:44 2018 calling bwa for /shen/shenlabstore2/sethberg/projects/hHumanFetal/fastp_output/IJ052.fastp.100bp.R1.fastq.gz Tue Nov 20 14:04:01 2018 calling bwa for /shen/shenlabstore2/sethberg/projects/hHumanFetal/fastp_output/IJ052.fastp.100bp.R2.fastq.gz Tue Nov 20 15:26:08 2018 calling samtools sort for /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/tempfiles/IJ052.fastp.100bp.R1.fastq.gz.bwa.sam storing in /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/tempfiles/IJ052.fastp.100bp.R1.fastq.gz.bwa.sam.srtn Tue Nov 20 15:45:09 2018 calling samtools sort for /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/tempfiles/IJ052.fastp.100bp.R2.fastq.gz.bwa.sam storing in /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/tempfiles/IJ052.fastp.100bp.R2.fastq.gz.bwa.sam.srtn [W::sam_read1] Parse error at line 13480925 Traceback (most recent call last): File "/netapp/home/sethberg/dependencies/MAPS/bin/feather/feather_pipe", line 116, in main() File "/netapp/home/sethberg/dependencies/MAPS/bin/feather/feather_pipe", line 49, in main filter_output_filename = filter_main(fastq1, fastq2, bwa_index, mapq, outdir, prefix, threads, to_file = False) File "/netapp/home/sethberg/dependencies/MAPS/bin/feather/feather_filter_chr.py", line 53, in filter_main pysam.sort("-o", bwa_sorted_filename , "-n", "-@", str(threads), bwa_filename) File "/netapp/home/sethberg/dependencies/Python-3.4.9/install_madebyseth/lib/python3.4/site-packages/pysam/utils.py", line 75, in call stderr)) pysam.utils.SamtoolsError: 'samtools returned with error 1: stdout=, stderr=samtools sort: truncated file. Aborting\n' sed: can't read /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/IJ052.feather.qc: No such file or directory

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ijuric/MAPS/issues/1, or mute the thread https://github.com/notifications/unsubscribe-auth/ALPpZbmRUZCwBTpVz9gdwCiNThPbKbn0ks5uxJjKgaJpZM4YsJvp .

armenabnousi commented 5 years ago

Hi Seth,

Since pysam.sort's error is "truncated file", I'd think the original error has happened during the BWA (or maybe something at the end of your fastq file?). Could you please check the last few lines of these two bwa output and bwa log file and have post them here. These commands should give you the last 20 lines of those files:

tail -n 20 /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/tempfiles/IJ052.fastp.100bp.R2.fastq.gz.bwa.sam

tail -n20 /shen/shenlabstore2/sethberg/projects/hHumanFetal/feather_output/feather_output/IJ052_20181120_093243/tempfiles/IJ052.fastp.100bp.R2.fastq.gz.bwa.sam.log

Thanks!