gbouras13
commented
8 months ago Hi @monty357penguin ,
This looks like a GPU memory issue with medaka.
Try this to turn off your GPU(s) before running hybracter
export CUDA_VISIBLE_DEVICES=""then your hybracter hybrid-single command
Hopefully that works.
George
monty357penguin
Hello George. When I try hybracter hybrid-single, it seems to stop with medaka consensus step. It works when I add --no_medaka.
Error log from medaka is as follows.
Creating fai index file /home/monty/Desktop/NGS_rawdata/NGS_rawdata/hybracterMb1/processing/complete/dnaapler/Mb1/Mb1_reoriented_chrom_and_plasmids.fasta.fai Creating mmi index file /home/monty/Desktop/NGS_rawdata/NGS_rawdata/hybracterMb1/processing/complete/dnaapler/Mb1/Mb1_reoriented_chrom_and_plasmids.fasta.map-ont.mmi [M::mm_idx_gen::0.061*0.51] collected minimizers [M::mm_idx_gen::0.103*0.45] sorted minimizers [M::main::0.114*0.50] loaded/built the index for 1 target sequence(s) [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1 [M::mm_idx_stat::0.115*0.51] distinct minimizers: 165648 (95.83% are singletons); average occurrences: 1.136; average spacing: 5.340; total length: 1005302 [M::main] Version: 2.27-r1193 [M::main] CMD: minimap2 -I 16G -x map-ont -d /home/monty/Desktop/NGS_rawdata/NGS_rawdata/hybracterMb1/processing/complete/dnaapler/Mb1/Mb1_reoriented_chrom_and_plasmids.fasta.map-ont.mmi /home/monty/Desktop/NGS_rawdata/NGS_rawdata/hybracterMb1/processing/complete/dnaapler/Mb1/Mb1_reoriented_chrom_and_plasmids.fasta [M::main] Real time: 0.117 sec; CPU: 0.061 sec; Peak RSS: 0.015 GB [M::main::0.013*1.08] loaded/built the index for 1 target sequence(s) [M::mm_mapopt_update::0.031*0.52] mid_occ = 19 [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1 [M::mm_idx_stat::0.033*0.54] distinct minimizers: 165648 (95.83% are singletons); average occurrences: 1.136; average spacing: 5.340; total length: 1005302 [M::worker_pipeline::8.489*3.55] mapped 30134 sequences [M::main] Version: 2.27-r1193 [M::main] CMD: minimap2 -x map-ont --secondary=no -L --MD -A 2 -B 4 -O 4,24 -E 2,1 -t 12 -a /home/monty/Desktop/NGS_rawdata/NGS_rawdata/hybracterMb1/processing/complete/dnaapler/Mb1/Mb1_reoriented_chrom_and_plasmids.fasta.map-ont.mmi /home/monty/Desktop/NGS_rawdata/NGS_rawdata/hybracterMb1/processing/qc/Mb1_filt_trim.fastq.gz [M::main] Real time: 8.491 sec; CPU: 30.124 sec; Peak RSS: 0.643 GB [bam_sort_core] merging from 0 files and 12 in-memory blocks... [14:55:45 - Predict] Reducing threads to 2, anymore is a waste. [14:55:45 - Predict] Setting tensorflow inter/intra-op threads to 2/1. [14:55:45 - Predict] Processing region(s): contig_3:0-1005302 [14:55:45 - Predict] Using model: /home/monty/miniforge3/envs/hybracterENV/lib/python3.12/site-packages/hybracter/workflow/conda/3da70fa7ba64389247470f11bcf39887_/lib/python3.10/site-packages/medaka/data/r1041_e82_400bps_sup_v4.2.0_model.tar.gz. [14:55:45 - Predict] Found a GPU. [14:55:45 - Predict] If cuDNN errors are observed, try setting the environment variableTF_FORCE_GPU_ALLOW_GROWTH=true. To explicitely disable use of cuDNN use the commandline option--disable_cudnn. If OOM (out of memory) errors are found please reduce batch size. [14:55:45 - Predict] Processing 2 long region(s) with batching. [14:55:45 - ModelLoad] GPU available: building model with cudnn optimization [14:55:46 - MdlStrTF] Model <keras.engine.sequential.Sequential object at 0x7cad225ced70> [14:55:46 - MdlStrTF] loading weights from /tmp/tmplmfa061n/model/variables/variables [14:55:46 - BAMFile] Creating pool of 16 BAM file sets. [14:55:46 - Sampler] Initializing sampler for consensus of region contig_3:0-1000000. [14:55:46 - Sampler] Initializing sampler for consensus of region contig_3:999000-1005302. [14:55:46 - PWorker] Running inference for 1.0M draft bases. [14:55:46 - Feature] Processed contig_3:999000.0-1005301.0 (median depth 37.0) [14:55:46 - Sampler] Took 0.26s to make features. [14:55:46 - Sampler] Region contig_3:999000.0-1005301.0 (9574 positions) is smaller than inference chunk length 10000, quarantining. [14:55:49 - Feature] Processed contig_3:0.0-999999.1 (median depth 92.0) [14:55:49 - Sampler] Took 2.66s to make features. 2024-03-27 14:56:00.264341: E tensorflow/stream_executor/dnn.cc:868] OOM when allocating tensor with shape[768000004] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc 2024-03-27 14:56:10.265309: E tensorflow/stream_executor/dnn.cc:868] OOM when allocating tensor with shape[768000004] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [14:56:10 - MdlStrTF] ModelStoreTF exception <class 'tensorflow.python.framework.errorsimpl.InternalError'> Traceback (most recent call last): File "/home/monty/miniforge3/envs/hybracterENV/lib/python3.12/site-packages/hybracter/workflow/conda/3da70fa7ba64389247470f11bcf39887/bin/medaka", line 11, inFailed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 3, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 256, 128, 1, 10000, 100, 0] [[{{node CudnnRNN}}]] [[sequential/bidirectional_1/backward_gru2/PartitionedCall]] [Op:__inference_predict_function_3261]`