search

single-cell-genetics / cellsnp-lite

Efficient genotyping bi-allelic SNPs on single cells
https://cellsnp-lite.readthedocs.io
Apache License 2.0
124 stars 11 forks source link

[E::bgzf_flush] File write failed (wrong size) #117

Closed ghuls closed 3 months ago

ghuls commented 4 months ago 
...
[I::csp_fetch_core][Thread-18] 26.00% SNPs processed.                                                                                                                                                                                       [1808/1881]
[I::csp_fetch_core][Thread-22] 52.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-13] 50.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-10] 44.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-21] 46.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-8] 52.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-0] 36.00% SNPs processed.                                                                       
[E::bgzf_flush] File write failed (wrong size)               
[I::csp_fetch_core][Thread-11] 60.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-4] 50.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-16] 64.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-5] 44.00% SNPs processed.                                                                       
[E::bgzf_flush] File write failed (wrong size)               
[I::csp_fetch_core][Thread-18] 28.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-9] 52.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-23] 70.00% SNPs processed.                                                                      
[E::bgzf_flush] File write failed (wrong size)               
[I::csp_fetch_core][Thread-14] 44.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-15] 50.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-22] 54.00% SNPs processed.                                                                      
[E::bgzf_flush] File write failed (wrong size)               
[I::csp_fetch_core][Thread-3] 56.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-21] 48.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-12] 48.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-17] 42.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-7] 58.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-2] 44.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-19] 34.00% SNPs processed.                                                                      
[E::bgzf_flush] File write failed (wrong size)               
[I::csp_fetch_core][Thread-6] 70.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-1] 46.00% SNPs processed.                                                                       
[E::bgzf_flush] File write failed (wrong size)               
[I::csp_fetch_core][Thread-11] 62.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-13] 52.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-18] 30.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-23] 72.00% SNPs processed.                                                                      
[E::bgzf_flush] File write failed (wrong size)               
[I::csp_fetch_core][Thread-16] 66.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-15] 52.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-10] 46.00% SNPs processed.                                                                      
[E::bgzf_flush] File write failed (wrong size)               
[I::csp_fetch_core][Thread-3] 58.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-22] 56.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-12] 50.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-9] 54.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-17] 44.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-21] 50.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-14] 46.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-1] 48.00% SNPs processed.                                                                       
[E::bgzf_flush] File write failed (wrong size)               
[I::csp_fetch_core][Thread-0] 38.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-20] 24.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-2] 46.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-11] 64.00% SNPs processed.                                                                      
[I::csp_fetch_core][Thread-7] 60.00% SNPs processed.                                                                       
[I::csp_fetch_core][Thread-18] 32.00% SNPs processed. 
...
[I::csp_fetch_core][Thread-20] 78.00% SNPs processed.
[I::csp_fetch_core][Thread-18] 94.00% SNPs processed.
[I::csp_fetch_core][Thread-18] 96.00% SNPs processed.
[I::csp_fetch_core][Thread-20] 80.00% SNPs processed.
[I::csp_fetch_core][Thread-18] 98.00% SNPs processed.
[I::csp_fetch_core][Thread-20] 82.00% SNPs processed.
[I::csp_fetch_core][Thread-20] 84.00% SNPs processed.
[I::csp_fetch_core][Thread-20] 86.00% SNPs processed.
[I::csp_fetch_core][Thread-20] 88.00% SNPs processed.
[I::csp_fetch_core][Thread-20] 90.00% SNPs processed.
[I::csp_fetch_core][Thread-20] 92.00% SNPs processed.
[I::csp_fetch_core][Thread-20] 94.00% SNPs processed.
[I::csp_fetch_core][Thread-20] 96.00% SNPs processed.
[I::csp_fetch_core][Thread-20] 98.00% SNPs processed.
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::bgzf_uncompress] Inflate operation failed: 1
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::bgzf_uncompress] Inflate operation failed: 1
[E::bgzf_uncompress] Inflate operation failed: 1
[E::csp_fetch] failed to merge mtx DP.
[E::main] running mode 1a failed.
[E::main] Quiting...
[I::main] Version: 1.2.3 (htslib 1.19)
[I::main] CMD: cellsnp-lite -s PB-002-pool_7.sorted.bam -O cellsnp_parsebio/ -R 1000g_hg38_high_cov_snps_af_0.1_to_0.9.liftover_stable.hg38.bcf -b parsebio_selected_barcodes/PB-002-pool_7.txt --UMItag pN --gzip -p 24
[I::main] end time: 2024-02-23 10:31:42
[I::main] time spent: 60288 seconds.

Input files seem to be OK:

# Decompressing the BCF file (5.6G) with bgzip, does not give any errors:
$ bgzip -c -d 1000g_hg38_high_cov_snps_af_0.1_to_0.9.liftover_stable.hg38.bcf  > /dev/null ; echo $?
0

# Decompressing the BAM file with bgzip, does not give any errors:
$ bgzip -c -d PB-002-pool_7.sorted.bam > /dev/null ; echo $?
0

Are the bgzf_flush errors from writing partial cellSNP.base.vcf.gz.0, cellSNP.base.vcf.gz.1 files?

So far the same command worked on other BAM files. Only PB-002-pool_5.sorted.bam and PB-002-pool_7.sorted.bam gave those errors (PB-002-pool_9.sorted.bam and PB-002-pool_10.sorted.bam are still running). By chance (or not), those are also the biggest files. Any idea what could be going on? Like an integer overflow.

PB-002-pool_1.sorted.bam        11.2198G
PB-002-pool_10.sorted.bam       45.1524G
PB-002-pool_2.sorted.bam        20.322G
PB-002-pool_3.sorted.bam        71.8872G
PB-002-pool_4.sorted.bam        44.142G
PB-002-pool_5.sorted.bam        111.943G
PB-002-pool_6.sorted.bam        79.5029G
PB-002-pool_7.sorted.bam        91.9134G
PB-002-pool_8.sorted.bam        54.821G
PB-002-pool_9.sorted.bam        35.943G
hxj5 commented 4 months ago

Hi, thanks for the feedback and checking the input files. The input files should be fine and the issue seems a htslib (bgzf_flush) error. After google search, I found three related issues on Github indicating that the error could be caused by insufficient disk space, given the large size of the input BAM file. You may see the three issues for detailed discussion: issue-1, issue-2, and issue-3.

You may check your disk and try increasing the quota. Alternatively, you can reduce the number of threads (-p option) in cellsnp-lite to save memory in case there is a out-of-memory problem, as this Biostars page indicates.

ghuls commented 4 months ago

It was indeed a quota problem and I checked the quota before reporting this error. Another user filled up the quota and deleted some files when they saw it, so when I checked the quota, there was space again (more than 1 TB) , so I didn't expect disk space issues.