[E::fai_retrieve] Failed to retrieve block: unexpected end of file

[JasonNPitt]

Describe the bug I'm getting a memory/file bug that only occurs some of the time when running medaka_consensus /// root@c8560012298f:~/03_assembly_data/PIPELINE_TEST_3/barcode_02/split_2# medaka_consensus -t 2 -m /home/ubuntu/r1041_e82_260bps_fast_g632_model.tar.gz -i /home/ubuntu/03_assembly_data/PIPELINE_TEST_3/ barcode_02/filtered.fastq -d /home/ubuntu/03_assembly_data/PIPELINE_TEST_3/barcode_02/rough_assembly.fasta -o /home/ubuntu/03_assembly_data/PIPELINE_TEST_3/barcode_02/split_2/temp_file_for_medaka_out Checking program versions This is medaka 1.8.1 Program Version Required Pass
bcftools 1.13 1.11 True
bgzip 1.13+ds 1.11 True
minimap2 2.24 2.11 True
samtools 1.17 1.11 True
tabix 1.13+ds 1.11 True
Warning: Output /home/ubuntu/03_assembly_data/PIPELINE_TEST_3/barcode_02/split_2/temp_file_for_medaka_out already exists, may use old results. Aligning basecalls to draft Using the existing fai index file /home/ubuntu/03_assembly_data/PIPELINE_TEST_3/barcode_02/rough_assembly.fasta.fai Using the existing mmi index file /home/ubuntu/03_assembly_data/PIPELINE_TEST_3/barcode_02/rough_assembly.fasta.map-ont.mmi [M::main::0.0031.27] loaded/built the index for 1 target sequence(s) [M::mm_mapopt_update::0.0031.26] mid_occ = 15 [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1 [M::mm_idx_stat::0.0031.24] distinct minimizers: 2476 (30.74% are singletons); average occurrences: 1.744; average spacing: 5.382; total length: 23246 [M::worker_pipeline::14.4291.76] mapped 3068 sequences [M::main] Version: 2.24-r1122 [M::main] CMD: minimap2 -x map-ont --secondary=no -L --MD -A 2 -B 4 -O 4,24 -E 2,1 -t 2 -a /home/ubuntu/03_assembly_data/PIPELINE_TEST_3/barcode_02/rough_assembly.fasta.map-ont.mmi /home/ubuntu/03_assembly_data/PIPELINE_TEST_3/barcode_02/filtered.fastq [M::main] Real time: 14.433 sec; CPU: 25.357 sec; Peak RSS: 0.134 GB [bam_sort_core] merging from 0 files and 2 in-memory blocks... Running medaka consensus [01:39:49 - Predict] Setting tensorflow inter/intra-op threads to 2/1. [01:39:49 - Predict] Processing region(s): tig00000001:0-23246 [01:39:49 - Predict] Using model: /home/ubuntu/r1041_e82_260bps_fast_g632_model.tar.gz. 2023-07-15 01:39:49.592672: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected [01:39:49 - Predict] Processing 1 long region(s) with batching. [01:39:50 - MdlStrTF] Model <keras.engine.sequential.Sequential object at 0x7fd913d91ed0> [01:39:50 - MdlStrTF] loading weights from /tmp/tmpt8am8ux8/model/variables/variables [01:39:50 - BAMFile] Creating pool of 16 BAM file sets. [01:39:50 - Sampler] Initializing sampler for consensus of region tig00000001:0-23246. [01:39:50 - PWorker] Running inference for 0.0M draft bases. [01:39:55 - Feature] Processed tig00000001:0.0-23245.0 (median depth 1666.0) [01:39:55 - Sampler] Took 4.64s to make features. [01:39:59 - PWorker] Batches in cache: 1. [01:39:59 - PWorker] Processed 1 batches [01:39:59 - PWorker] All done, 0 remainder regions. [01:39:59 - Predict] Finished processing all regions. [01:40:02 - DataIndx] Loaded 1/1 (100.00%) sample files. draft pre = /home/ubuntu/03_assembly_data/PIPELINE_TEST_3/barcode_02/rough_assembly.fasta draft post = <pysam.libcfaidx.FastaFile object at 0x7f8779fdce80> [01:40:02 - DataIndx] Loaded 1/1 (100.00%) sample files. medaka ref name = tig00000001 [E::fai_retrieve] Failed to retrieve block: unexpected end of file Traceback (most recent call last): File "/usr/local/bin/medaka", line 8, in sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/medaka/medaka.py", line 724, in main args.func(args) File "/usr/local/lib/python3.10/dist-packages/medaka/stitch.py", line 232, in stitch contigs, gt = fill_gaps(contigs, args.draft, args.fill_char) File "/usr/local/lib/python3.10/dist-packages/medaka/stitch.py", line 131, in fill_gaps draft_seq = draft.fetch(ref_name) File "pysam/libcfaidx.pyx", line 317, in pysam.libcfaidx.FastaFile.fetch ValueError: failure when retrieving sequence on 'tig00000001' Failed to stitch consensus chunks. ///

**Environment: No GPU

• Installation method: pip3
• OS: Ubuntu 22.04
• medaka version: 1.8.1

Additional context This bug happens only about 33% of the time I run medaka. The draft pre, draft post, and medaka ref lines are some debugging I added to the fill_gaps function in the stitch.py source to try to figure out why this is crashing. The fasta file is valid and I don't understand why pysam is throwing this error. This is running in a Docker container not conda

[JasonNPitt]

after further library adjustments I think this is a pysam/samtools issue

[tibitoy]

Hi @JasonNPitt ! Can you share how you remedied this issue? I'm encountering the same thing. It only occurs some of the time.