Failed to run medaka consensus

[Sabarish2001]

Hello, i tried running medaka_consensus and i am resulting in the following error. I installed medaka through conda package manager.

The command i used to run medaka is the following :

medaka_consensus -i trycycler/cluster_001/4_reads.fastq -d trycycler/cluster_001/7_final_consensus.fasta -o trycycler/cluster_001/medaka -m r104_e81_sup_g610

I am resulting in the following error:

TF_CPP_MIN_LOG_LEVEL is set to '3' Checking program versions This is medaka 1.11.3 Program Version Required Pass bcftools 1.17 1.11 True bgzip 1.17 1.11 True minimap2 2.28 2.11 True samtools 1.18 1.11 True tabix 1.17 1.11 True [11:56:32 - MdlStrTF] Successfully removed temporary files from /tmp/tmp2s_rjl9y. [11:56:32 - MdlStrTF] Successfully removed temporary files from /tmp/tmp5fmjjlro. Aligning basecalls to draft Creating fai index file /home/sabioinfo/hands_on_assembly/trycycler/cluster_001/7_final_consensus.fasta.fai Creating mmi index file /home/sabioinfo/hands_on_assembly/trycycler/cluster_001/7_final_consensus.fasta.map-ont.mmi [M::mm_idx_gen::0.0580.99] collected minimizers [M::mm_idx_gen::0.0761.31] sorted minimizers [M::main::0.0921.25] loaded/built the index for 1 target sequence(s) [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1 [M::mm_idx_stat::0.0991.23] distinct minimizers: 506947 (97.56% are singletons); average occurrences: 1.040; average spacing: 5.345; total length: 2818314 [M::main] Version: 2.28-r1209 [M::main] CMD: minimap2 -I 16G -x map-ont -d /home/sabioinfo/hands_on_assembly/trycycler/cluster_001/7_final_consensus.fasta.map-ont.mmi /home/sabioinfo/hands_on_assembly/trycycler/cluster_001/7_final_consensus.fasta [M::main] Real time: 0.103 sec; CPU: 0.126 sec; Peak RSS: 0.031 GB [M::main::0.0241.03] loaded/built the index for 1 target sequence(s) [M::mm_mapopt_update::0.0311.02] mid_occ = 10 [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1 [M::mm_idx_stat::0.0361.02] distinct minimizers: 506947 (97.56% are singletons); average occurrences: 1.040; average spacing: 5.345; total length: 2818314 [M::worker_pipeline::157.7330.86] mapped 37504 sequences [M::worker_pipeline::322.1320.86] mapped 37008 sequences [M::worker_pipeline::484.1030.86] mapped 37129 sequences [M::worker_pipeline::564.619*0.87] mapped 20441 sequences [M::main] Version: 2.28-r1209 [M::main] CMD: minimap2 -x map-ont --secondary=no -L --MD -A 2 -B 4 -O 4,24 -E 2,1 -t 1 -a /home/sabioinfo/hands_on_assembly/trycycler/cluster_001/7_final_consensus.fasta.map-ont.mmi /home/sabioinfo/hands_on_assembly/trycycler/cluster_001/4_reads.fastq [M::main] Real time: 564.629 sec; CPU: 491.228 sec; Peak RSS: 1.661 GB [bam_sort_core] merging from 3 files and 1 in-memory blocks... Running medaka consensus [12:07:06 - Predict] Setting tensorflow inter/intra-op threads to 1/1. [12:07:06 - Predict] Processing region(s): cluster_001_consensus:0-2818314 [12:07:06 - Predict] Using model: /home/sabioinfo/miniconda3/envs/polishing/lib/python3.10/site-packages/medaka/data/r104_e81_sup_g610_model.tar.gz. [12:07:07 - Predict] Found a GPU. [12:07:07 - Predict] If cuDNN errors are observed, try setting the environment variable TF_FORCE_GPU_ALLOW_GROWTH=true. To explicitely disable use of cuDNN use the commandline option `--disable_cudnn. If OOM (out of memory) errors are found please reduce batch size. [12:07:07 - BAMFile] Creating pool of 16 BAM file sets. [12:07:07 - Predict] Processing 3 long region(s) with batching. [12:07:07 - ModelLoad] GPU available: building model with cudnn optimization [12:07:09 - MdlStrTF] Model <keras.engine.sequential.Sequential object at 0x7f6166ac0100> [12:07:09 - MdlStrTF] loading weights from /tmp/tmpyymm80qz/model/variables/variables (using expect partial) [12:07:09 - Sampler] Initializing sampler for consensus of region cluster_001_consensus:0-1000000. [12:07:09 - Sampler] Initializing sampler for consensus of region cluster_001_consensus:999000-1999000. [12:07:09 - PWorker] Running inference for 2.8M draft bases. [12:07:23 - Feature] Processed cluster_001_consensus:999000.0-1998999.1 (median depth 551.0) [12:07:23 - Sampler] Took 14.06s to make features. [12:07:23 - Sampler] Initializing sampler for consensus of region cluster_001_consensus:1998000-2818314. [12:07:23 - Feature] Processed cluster_001_consensus:0.0-999999.1 (median depth 602.0) [12:07:23 - Sampler] Took 14.28s to make features. [12:07:30 - Feature] Processed cluster_001_consensus:1998000.0-2818313.0 (median depth 600.0) [12:07:30 - Sampler] Took 7.35s to make features. [12:07:43 - MdlStrTF] Successfully removed temporary files from /tmp/tmpyymm80qz. [12:07:43 - MdlStrTF] ModelStoreTF exception <class 'tensorflow.python.framework.errors_impl.InternalError'> Traceback (most recent call last): File "/home/sabioinfo/miniconda3/envs/polishing/bin/medaka", line 11, in sys.exit(main()) File "/home/sabioinfo/miniconda3/envs/polishing/lib/python3.10/site-packages/medaka/medaka.py", line 814, in main args.func(args) File "/home/sabioinfo/miniconda3/envs/polishing/lib/python3.10/site-packages/medaka/prediction.py", line 167, in predict remainder_regions_depth = run_prediction( File "/home/sabioinfo/miniconda3/envs/polishing/lib/python3.10/site-packages/medaka/prediction.py", line 47, in run_prediction class_probs = model.predict_on_batch(x_data) File "/home/sabioinfo/miniconda3/envs/polishing/lib/python3.10/site-packages/keras/engine/training.py", line 2474, in predict_on_batch outputs = self.predict_function(iterator) File "/home/sabioinfo/miniconda3/envs/polishing/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler raise e.with_traceback(filtered_tb) from None File "/home/sabioinfo/miniconda3/envs/polishing/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 3, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 10, 128, 1, 10000, 100, 0] [[{{node CudnnRNN}}]] [[sequential/bidirectional/backward_gru1/PartitionedCall]] [Op:__inference_predict_function_3295] Failed to run medaka consensus.

Can somebody shed some light on this?

[cjw85]

As stated in the README:

The bioconda medaka packages are no longer supported by Oxford Nanopore Technologies.

The recommended method of installation is through the packages on PyPI which are maintained and supported by us. If you still wish to use the conda package you will need to raise an issue on the bioconda recipes GitHub page.

[MauriAndresMU1313]

@cjw85 Thank you for providing that information. I removed medaka, which I previously installed using the conda way (bioconda), and I installed again using PyPI (pip install medaka). However, I still have the same error (with medaka installed by bioconda):

Command failed:medaka consensus --model r941_min_high_g351 --threads 2 --chunk_len 800 --chunk_ovlp 400 --RG Ebov-DRC_2 ebov-mayinga.trimmed.rg.sorted.bam ebov-mayinga.Ebov-DRC_2.hdf

If I check the version in my conda environment:

medaka                    1.11.3                   pypi_0    pypi

Could you provide more information about this issue? Is this issue related to this installation process? or is related to a specific issue with the sequences? I believe that is related to the sequences, however, this is the test process from this link: https://artic.readthedocs.io/en/latest/tests/?badge=latest After the installation, I performed the:

./test-runner.sh medaka
./test-runner.sh nanopolish

Then I got the error: failed to run medaka consensus Any information that you can provide is more than welcome! My Mac is Apple M2 max

[cjw85]

I'm sorry, I cannot comment on why medaka consensus has failed within a 3rd party pipeline. If you can provide a standalone, reproducible example with complete input and output then I will be able to help more.

[MauriAndresMU1313]

Thank you for the quick response, I will post the information as soon as possible