Failed to run medaka consensus. - ModelStoreTF exception <class 'NotImplementedError'>

Hello @rpetit3 ,

Thank you for developing dragonflye.

We are trying to use dragonflye to perform assembly on the E. faecium isolates with the following command: dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --nanohq --model r941_min_sup_g507

However, we encounted an error in the polishing step when using medaka. Please find the error logs below:

[dragonflye] Hello gilmansiu3 [dragonflye] You ran: /home/gilmansiu3/miniconda3/envs/dragonflye/bin/dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --nanohq --model r941_min_sup_g507 [dragonflye] This is dragonflye 1.0.13 [dragonflye] Written by Robert A Petit III [dragonflye] Homepage is https://github.com/rpetit3/dragonflye [dragonflye] Operating system is linux [dragonflye] Perl version is v5.32.1 [dragonflye] Machine has 20 CPU cores and 125.72 GB RAM [dragonflye] Verifying input model (--model): r941_min_sup_g507 [dragonflye] Model r941_min_sup_g507 verified! [dragonflye] Valid model provided, but number of Medaka rounds (--medaka) not given, assuming 1 round [dragonflye] Using any2fasta - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/any2fasta | any2fasta 0.4.2 [dragonflye] Using assembly-scan - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/assembly-scan | assembly-scan 0.4.1 [dragonflye] Using bwa - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/bwa | Version: 0.7.17-r1188 [dragonflye] Using fastp - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/fastp | fastp 0.23.2 [dragonflye] Using flye - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/flye | 2.9-b1768 [dragonflye] Using kmc - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/kmc | K-Mer Counter (KMC) ver. 3.2.1 (2022-01-04) [dragonflye] Using medaka - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/medaka | medaka 1.6.1 [dragonflye] Using miniasm - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/miniasm | 0.3-r179 [dragonflye] Using minimap2 - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/minimap2 | 2.24-r1122 [dragonflye] Using nanoq - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/nanoq | nanoq 0.9.0 [dragonflye] Using pigz - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/pigz | pigz 2.6 [dragonflye] Using pilon - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/pilon | Pilon version 1.24 Thu Jan 28 13:00:45 2021 -0500 [dragonflye] Using polypolish - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/polypolish | Polypolish v0.5.0 [dragonflye] Using porechop - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/porechop | 0.2.4 [dragonflye] Using racon - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/racon | 1.5.0 [dragonflye] Using rasusa - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/rasusa | rasusa 0.7.0 [dragonflye] Using raven - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/raven | 1.8.1 [dragonflye] Using samclip - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/samclip | samclip 0.4.0 [dragonflye] Using samtools - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/samtools | Version: 1.15.1 (using htslib 1.15.1) [dragonflye] Using seqtk - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/seqtk | Version: 1.3-r106 [dragonflye] Using tempdir: /tmp/tXf0FpvLDL [dragonflye] Changing into folder: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1 [dragonflye] Collecting raw read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 \/mnt\/data\/Species-specific\/CAUR\/02_fastq\/220818_VRE1.fastq.gz 2>&1 1>/tmp/deywkg47eV | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Read stats: avg_len = 4267 [dragonflye] Read stats: max_len = 45857 [dragonflye] Read stats: min_len = 1000 [dragonflye] Read stats: total_bp = 432559422 [dragonflye] Using genome size 2800000 bp [dragonflye] Estimated sequencing depth: 154x [dragonflye] Filter reads based on length and/or quality [dragonflye] Running: nanoq --min-len 1000 --input /mnt/data/Species-specific/CAUR/02_fastq/220818_VRE1.fastq.gz --min-qual 0 2>&1 1> READS.filt.fq | sed 's/^/[nanoq] /' | tee -a dragonflye.log [dragonflye] Running: pigz -f -p 20 --fast READS.filt.fq 2>&1 | sed 's/^/[pigz] /' | tee -a dragonflye.log [dragonflye] No read depth reduction requested or necessary. [dragonflye] No read adapter trimming requested. [dragonflye] Running: ln -sf READS.filt.fq.gz READS.fq.gz 2>&1 | sed 's/^/[ln] /' | tee -a dragonflye.log [dragonflye] Collecting qc'd read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 READS.fq.gz 2>&1 1>/tmp/l7Duy6iujE | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Final Read stats: min_len = 1000 [dragonflye] Final Read stats: max_len = 45857 [dragonflye] Final Read stats: avg_len = 4267 [dragonflye] Final Read stats: total_bp = 432559422 [dragonflye] Average read length looks like 4267 bp [dragonflye] Assembling reads with 'flye' [dragonflye] Running: flye --nano-hq READS.fq.gz -g 2800000 -i 0 --threads 20 -o flye 2>&1 | sed 's/^/[flye] /' | tee -a dragonflye.log [flye] [2022-11-25 15:10:17] INFO: Starting Flye 2.9-b1768 [flye] [2022-11-25 15:10:17] INFO: >>>STAGE: configure [flye] [2022-11-25 15:10:17] INFO: Configuring run [flye] [2022-11-25 15:10:22] INFO: Total read length: 432559422 [flye] [2022-11-25 15:10:22] INFO: Input genome size: 2800000 [flye] [2022-11-25 15:10:22] INFO: Estimated coverage: 154 [flye] [2022-11-25 15:10:22] INFO: Reads N50/N90: 5835 / 2000 [flye] [2022-11-25 15:10:22] INFO: Minimum overlap set to 2000 [flye] [2022-11-25 15:10:22] INFO: >>>STAGE: assembly [flye] [2022-11-25 15:10:22] INFO: Assembling disjointigs [flye] [2022-11-25 15:10:22] INFO: Reading sequences [flye] [2022-11-25 15:10:27] INFO: Building minimizer index [flye] [2022-11-25 15:10:27] INFO: Pre-calculating index storage [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-11-25 15:10:30] INFO: Filling index [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-11-25 15:10:39] INFO: Extending reads [flye] [2022-11-25 15:11:21] INFO: Overlap-based coverage: 111 [flye] [2022-11-25 15:11:21] INFO: Median overlap divergence: 0.0539394 [flye] 0% 10% 90% 100% [flye] [2022-11-25 15:12:13] INFO: Assembled 6 disjointigs [flye] [2022-11-25 15:12:13] INFO: Generating sequence [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-11-25 15:12:14] INFO: Filtering contained disjointigs [flye] 0% 10% 30% 50% 60% 80% 100% [flye] [2022-11-25 15:12:14] INFO: Contained seqs: 0 [flye] [2022-11-25 15:12:14] INFO: >>>STAGE: consensus [flye] [2022-11-25 15:12:14] INFO: Running Minimap2 [flye] [2022-11-25 15:12:43] INFO: Computing consensus [flye] [2022-11-25 15:14:22] INFO: Alignment error rate: 0.067847 [flye] [2022-11-25 15:14:22] INFO: >>>STAGE: repeat [flye] [2022-11-25 15:14:22] INFO: Building and resolving repeat graph [flye] [2022-11-25 15:14:22] INFO: Parsing disjointigs [flye] [2022-11-25 15:14:22] INFO: Building repeat graph [flye] 0% 10% 30% 50% 60% 80% 100% [flye] [2022-11-25 15:14:23] INFO: Median overlap divergence: 0.00335946 [flye] [2022-11-25 15:14:23] INFO: Parsing reads [flye] [2022-11-25 15:14:27] INFO: Aligning reads to the graph [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-11-25 15:14:36] INFO: Aligned read sequence: 383411070 / 389252928 (0.984992) [flye] [2022-11-25 15:14:36] INFO: Median overlap divergence: 0.0258973 [flye] [2022-11-25 15:14:36] INFO: Mean edge coverage: 122 [flye] [2022-11-25 15:14:36] INFO: Simplifying the graph [flye] [2022-11-25 15:14:36] INFO: >>>STAGE: contigger [flye] [2022-11-25 15:14:36] INFO: Generating contigs [flye] [2022-11-25 15:14:36] INFO: Reading sequences [flye] [2022-11-25 15:14:41] INFO: Generated 7 contigs [flye] [2022-11-25 15:14:41] INFO: Added 0 scaffold connections [flye] [2022-11-25 15:14:41] INFO: >>>STAGE: finalize [flye] [2022-11-25 15:14:41] INFO: Assembly statistics: [flye] [flye] Total length: 3161882 [flye] Fragments: 7 [flye] Fragments N50: 2791738 [flye] Largest frg: 2791738 [flye] Scaffolds: 0 [flye] Mean coverage: 121 [flye] [flye] [2022-11-25 15:14:41] INFO: Final assembly: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/assembly.fasta [dragonflye] Polishing with Racon (1 rounds) [dragonflye] Running: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz 2>&1 1> flye/polish/racon/1/aligments.paf | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0551.01] collected minimizers [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0612.58] sorted minimizers [polishing - racon (1 of 1)] [M::main::0.0612.58] loaded/built the index for 7 target sequence(s) [polishing - racon (1 of 1)] [M::mm_mapopt_update::0.0672.45] mid_occ = 26 [polishing - racon (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 7 [polishing - racon (1 of 1)] [M::mm_idx_stat::0.0712.36] distinct minimizers: 534042 (95.78% are singletons); average occurrences: 1.109; average spacing: 5.339; total length: 3161882 [polishing - racon (1 of 1)] [M::worker_pipeline::11.02911.97] mapped 101377 sequences [polishing - racon (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - racon (1 of 1)] [M::main] CMD: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz [polishing - racon (1 of 1)] [M::main] Real time: 11.035 sec; CPU: 132.074 sec; Peak RSS: 0.491 GB [dragonflye] Running: racon -t 20 READS.fq.gz flye/polish/racon/1/aligments.paf flye.fasta 2>&1 1> flye/polish/racon/1/consensus.fasta | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded target sequences 0.012280 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded sequences 4.787064 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded overlaps 0.085765 s [racon::Polisher::initialize] aligning overlaps [====================] 8.314751 s ] 0.522070 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] transformed data into windows 0.466438 s [racon::Polisher::polish] generating consensus [====================] 44.492387 s ] 3.252132 s [polishing - racon (1 of 1)] [racon::Polisher::] total = 58.222555 s [dragonflye] Polishing with Medaka (1 rounds) [dragonflye] Running: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log [polishing - medaka (1 of 1)] Checking program versions [polishing - medaka (1 of 1)] This is medaka 1.6.1 [polishing - medaka (1 of 1)] Program Version Required Pass
[polishing - medaka (1 of 1)] bcftools 1.15.1 1.11 True
[polishing - medaka (1 of 1)] bgzip 1.15.1 1.11 True
[polishing - medaka (1 of 1)] minimap2 2.24 2.11 True
[polishing - medaka (1 of 1)] samtools 1.15.1 1.11 True
[polishing - medaka (1 of 1)] tabix 1.15.1 1.11 True
[polishing - medaka (1 of 1)] Aligning basecalls to draft [polishing - medaka (1 of 1)] Creating fai index file /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.fai [polishing - medaka (1 of 1)] Creating mmi index file /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.1031.02] collected minimizers [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.1131.19] sorted minimizers [polishing - medaka (1 of 1)] [M::main::0.1291.17] loaded/built the index for 7 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 7 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.1331.16] distinct minimizers: 529319 (95.23% are singletons); average occurrences: 1.119; average spacing: 5.339; total length: 3163172 [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -I 16G -x map-ont -d /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta [polishing - medaka (1 of 1)] [M::main] Real time: 0.135 sec; CPU: 0.157 sec; Peak RSS: 0.033 GB [polishing - medaka (1 of 1)] [M::main::0.0191.03] loaded/built the index for 7 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_mapopt_update::0.0241.02] mid_occ = 27 [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 7 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.0281.02] distinct minimizers: 529319 (95.23% are singletons); average occurrences: 1.119; average spacing: 5.339; total length: 3163172 [polishing - medaka (1 of 1)] [M::worker_pipeline::21.05013.97] mapped 101377 sequences [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -x map-ont --secondary=no -L --MD -A 2 -B 4 -O 4,24 -E 2,1 -t 20 -a /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/READS.filt.fq.gz [polishing - medaka (1 of 1)] [M::main] Real time: 21.053 sec; CPU: 294.060 sec; Peak RSS: 1.828 GB [polishing - medaka (1 of 1)] [bam_sort_core] merging from 0 files and 20 in-memory blocks... [polishing - medaka (1 of 1)] Running medaka consensus [polishing - medaka (1 of 1)] [15:16:26 - Predict] Reducing threads to 2, anymore is a waste. [polishing - medaka (1 of 1)] [15:16:27 - Predict] Setting tensorflow inter/intra-op threads to 2/1. [polishing - medaka (1 of 1)] [15:16:27 - Predict] Processing region(s): contig_1:0-235990 contig_2:0-2793735 contig_3:0-45936 contig_4:0-8983 contig_5:0-32435 contig_6:0-34261 contig_7:0-11832 [polishing - medaka (1 of 1)] [15:16:27 - Predict] Using model: /home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/data/r941_min_sup_g507_model.tar.gz. [polishing - medaka (1 of 1)] [15:16:27 - Predict] Found a GPU. [polishing - medaka (1 of 1)] [15:16:27 - Predict] If cuDNN errors are observed, try setting the environment variable TF_FORCE_GPU_ALLOW_GROWTH=true. To explicitely disable use of cuDNN use the commandline option `--disable_cudnn. If OOM (out of memory) errors are found please reduce batch size. [polishing - medaka (1 of 1)] [15:16:27 - Predict] Processing 9 long region(s) with batching. [polishing - medaka (1 of 1)] [15:16:27 - ModelLoad] GPU available: building model with cudnn optimization [polishing - medaka (1 of 1)] [15:16:27 - MdlStrTF] ModelStoreTF exception <class 'NotImplementedError'> [polishing - medaka (1 of 1)] Traceback (most recent call last): [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/bin/medaka", line 11, in [polishing - medaka (1 of 1)] sys.exit(main()) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/medaka.py", line 720, in main [polishing - medaka (1 of 1)] args.func(args) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/prediction.py", line 160, in predict [polishing - medaka (1 of 1)] model = model_store.load_model(time_steps=args.chunk_len) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/datastore.py", line 159, in load_model [polishing - medaka (1 of 1)] self.model = model_partial_function(time_steps=time_steps) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/models.py", line 147, in build_model [polishing - medaka (1 of 1)] model.add(Bidirectional(gru, input_shape=input_shape)) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 456, in _method_wrapper [polishing - medaka (1 of 1)] result = method(self, args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py", line 198, in add [polishing - medaka (1 of 1)] layer(x) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/wrappers.py", line 531, in call [polishing - medaka (1 of 1)] return super(Bidirectional, self).call(inputs, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in call [polishing - medaka (1 of 1)] outputs = call_fn(cast_inputs, args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/wrappers.py", line 644, in call [polishing - medaka (1 of 1)] y = self.forward_layer(forward_inputs, [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 654, in call [polishing - medaka (1 of 1)] return super(RNN, self).call(inputs, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in call [polishing - medaka (1 of 1)] outputs = call_fn(cast_inputs, *args, *kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent_v2.py", line 408, in call [polishing - medaka (1 of 1)] inputs, initialstate, = self._process_inputs(inputs, initial_state, None) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 848, in _process_inputs [polishing - medaka (1 of 1)] initial_state = self.get_initial_state(inputs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 636, in get_initial_state [polishing - medaka (1 of 1)] init_state = get_initial_state_fn( [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 1910, in get_initial_state [polishing - medaka (1 of 1)] return _generate_zero_filled_state_for_cell(self, inputs, batch_size, dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2926, in _generate_zero_filled_state_for_cell [polishing - medaka (1 of 1)] return _generate_zero_filled_state(batch_size, cell.state_size, dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2944, in _generate_zero_filled_state [polishing - medaka (1 of 1)] return create_zeros(state_size) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2939, in create_zeros [polishing - medaka (1 of 1)] return array_ops.zeros(init_state_size, dtype=dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2677, in wrapped [polishing - medaka (1 of 1)] tensor = fun(args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2721, in zeros [polishing - medaka (1 of 1)] output = _constant_if_small(zero, shape, dtype, name) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2662, in _constant_if_small [polishing - medaka (1 of 1)] if np.prod(shape) < 1000: [polishing - medaka (1 of 1)] File "<__array_function__ internals>", line 180, in prod [polishing - medaka (1 of 1)] File "/home/gilmansiu3/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 3045, in prod [polishing - medaka (1 of 1)] return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out, [polishing - medaka (1 of 1)] File "/home/gilmansiu3/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction [polishing - medaka (1 of 1)] return ufunc.reduce(obj, axis, dtype, out, passkwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 748, in array [polishing - medaka (1 of 1)] raise NotImplementedError("Cannot convert a symbolic Tensor ({}) to a numpy" [polishing - medaka (1 of 1)] NotImplementedError: Cannot convert a symbolic Tensor (bidirectional/forward_gru1/strided_slice:0) to a numpy array. [polishing - medaka (1 of 1)] Failed to run medaka consensus. [dragonflye] Error running command: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log

Best regards, Eddie

Hi Eddie,

Do you mind sharing the output of conda env export in your dragonflye environment?

Thank you! Robert

The output are as below:

name: dragonflye channels:

conda-forge

bioconda

defaults

r

cruizperez

liulab-dfci dependencies:

_libgcc_mutex=0.1=conda_forge

_openmp_mutex=4.5=2_gnu

_tflow_select=2.1.0=gpu

abseil-cpp=20211102.0=h27087fc_1

absl-py=1.1.0=pyhd8ed1ab_0

aiohttp=3.8.1=py38h0a891b7_1

aiosignal=1.2.0=pyhd8ed1ab_0

alsa-lib=1.2.6.1=h7f98852_0

any2fasta=0.4.2=hdfd78af_3

assembly-scan=0.4.1=pyhdfd78af_0

astor=0.8.1=pyh9f0ad1d_0

astunparse=1.6.3=pyhd8ed1ab_0

async-timeout=4.0.2=pyhd8ed1ab_0

attrs=21.4.0=pyhd8ed1ab_0

bcftools=1.15.1=h0ea216a_0

biopython=1.79=py38h0a891b7_2

blinker=1.4=py_1

brotlipy=0.7.0=py38h0a891b7_1004

bwa=0.7.17=h7132678_9

bzip2=1.0.8=h7f98852_4

c-ares=1.18.1=h7f98852_0

ca-certificates=2022.6.15=ha878542_0

cachetools=5.0.0=pyhd8ed1ab_0

cairo=1.16.0=ha61ee94_1011

certifi=2022.6.15=py38h578d9bd_0

cffi=1.15.1=py38h4a40e3a_0

charset-normalizer=2.1.0=pyhd8ed1ab_0

click=8.1.3=py38h578d9bd_0

cryptography=37.0.4=py38h2b5fc30_0

cudatoolkit=10.1.243=h8cb64d8_10

cudnn=7.6.5.32=hc0a50b0_1

cupti=10.1.168=0

dragonflye=1.0.13=hdfd78af_0

expat=2.4.8=h27087fc_0

fastp=0.23.2=hb7a2d85_2

flye=2.9=py38hf4f3596_1

font-ttf-dejavu-sans-mono=2.37=hab24e00_0

font-ttf-inconsolata=3.000=h77eed37_0

font-ttf-source-code-pro=2.038=h77eed37_0

font-ttf-ubuntu=0.83=hab24e00_0

fontconfig=2.14.0=h8e229c2_0

fonts-conda-ecosystem=1=0

fonts-conda-forge=1=0

freetype=2.10.4=h0708190_1

frozenlist=1.3.0=py38h0a891b7_1

gast=0.3.3=py_0

gettext=0.19.8.1=h73d1719_1008

giflib=5.2.1=h36c2ea0_2

google-auth=2.9.1=pyh6c4a22f_0

google-auth-oauthlib=0.4.6=pyhd8ed1ab_0

google-pasta=0.2.0=pyh8c360ce_0

graphite2=1.3.13=h58526e2_1001

grpc-cpp=1.47.1=hbd84cd8_0

grpcio=1.47.1=py38h5436ba1_0

gsl=2.7=he838d99_0

h5py=2.10.0=nompi_py38h9915d05_106

harfbuzz=4.4.1=hf9f4e7c_0

hdf5=1.10.6=nompi_h6a2412b_1114

htslib=1.15.1=h9753748_0

icu=70.1=h27087fc_0

idna=3.3=pyhd8ed1ab_0

importlib-metadata=4.11.4=py38h578d9bd_0

intervaltree=3.0.2=py_0

isa-l=2.30.0=ha770c72_4

jpeg=9e=h166bdaf_2

k8=0.2.5=hd03093a_2

keras-preprocessing=1.1.2=pyhd8ed1ab_0

keyutils=1.6.1=h166bdaf_0

kmc=3.2.1=hf1761c0_2

krb5=1.19.3=h3790be6_0

lcms2=2.12=hddcbb42_0

ld_impl_linux-64=2.36.1=hea4e1c9_2

lerc=3.0=h9c3ff4c_0

libblas=3.9.0=15_linux64_openblas

libcblas=3.9.0=15_linux64_openblas

libcups=2.3.3=hf5a7f15_1

libcurl=7.83.1=h7bff187_0

libdeflate=1.10=h7f98852_0

libedit=3.1.20191231=he28a2e2_2

libev=4.33=h516909a_1

libffi=3.4.2=h7f98852_5

libgcc-ng=12.1.0=h8d9b700_16

libgfortran-ng=12.1.0=h69a702a_16

libgfortran5=12.1.0=hdcd56e2_16

libglib=2.72.1=h2d90d5f_0

libgomp=12.1.0=h8d9b700_16

libiconv=1.16=h516909a_0

liblapack=3.9.0=15_linux64_openblas

libnghttp2=1.47.0=h727a467_0

libnsl=2.0.0=h7f98852_0

libopenblas=0.3.20=pthreads_h78a6416_0

libpng=1.6.37=h753d276_3

libprotobuf=3.20.1=h6239696_0

libssh2=1.10.0=ha56f1ee_2

libstdcxx-ng=12.1.0=ha89aaad_16

libtiff=4.4.0=h0fcbabc_0

libuuid=2.32.1=h7f98852_1000

libwebp-base=1.2.2=h7f98852_1

libxcb=1.13=h7f98852_1004

libzlib=1.2.12=h166bdaf_2

lz4-c=1.9.3=h9c3ff4c_1

mappy=2.24=py38h4c6a040_1

markdown=3.4.1=pyhd8ed1ab_0

medaka=1.6.1=py38h84d2cc8_0

miniasm=0.3_r179=h7132678_3

minimap2=2.24=h7132678_1

multidict=6.0.2=py38h0a891b7_1

nanoq=0.9.0=hec16e2b_0

ncurses=6.3=h27087fc_1

networkx=2.8.4=pyhd8ed1ab_0

numpy=1.19.5=py38h8246c76_3

oauthlib=3.2.0=pyhd8ed1ab_0

ont-fast5-api=4.0.2=pyhdfd78af_0

openjdk=11.0.15=hc6918da_0

openssl=1.1.1q=h166bdaf_0

opt_einsum=3.3.0=pyhd8ed1ab_1

packaging=21.3=pyhd8ed1ab_0

parasail-python=1.2.4=py38h3b68952_2

pbzip2=1.1.13=0

pcre=8.45=h9c3ff4c_0

perl=5.32.1=2_h7f98852_perl5

perl-file-spec=3.48_01=pl5321hdfd78af_2

perl-findbin=1.51=pl5321hdfd78af_3

pigz=2.6=h27826a3_0

pilon=1.24=hdfd78af_0

pip=22.1.2=pyhd8ed1ab_0

pixman=0.40.0=h36c2ea0_0

polypolish=0.5.0=h87f3376_1

porechop=0.2.4=py38h4a32c8e_3

progressbar33=2.4=py_0

protobuf=3.20.1=py38hfa26641_0

pthread-stubs=0.4=h36c2ea0_1001

pyasn1=0.4.8=py_0

pyasn1-modules=0.2.7=py_0

pycparser=2.21=pyhd8ed1ab_0

pyfaidx=0.7.0=pyh5e36f6f_0

pyjwt=2.4.0=pyhd8ed1ab_0

pyopenssl=22.0.0=pyhd8ed1ab_0

pyparsing=3.0.9=pyhd8ed1ab_0

pysam=0.19.1=py38h8bf8b8d_0

pysocks=1.7.1=py38h578d9bd_5

pyspoa=0.0.3=py38h8ded8fe_3

python=3.8.13=h582c2e5_0_cpython

python-edlib=1.3.9=py38h4a32c8e_1

python-isal=0.11.1=py38h497a2fe_1

python_abi=3.8=2_cp38

pyu2f=0.1.5=pyhd8ed1ab_0

racon=1.5.0=h7ff8a90_0

rasusa=0.7.0=hec16e2b_0

raven-assembler=1.8.1=h5b5514e_0

re2=2022.06.01=h27087fc_0

readline=8.1.2=h0f457ee_0

requests=2.28.1=pyhd8ed1ab_0

requests-oauthlib=1.3.1=pyhd8ed1ab_0

rsa=4.8=pyhd8ed1ab_0

samclip=0.4.0=hdfd78af_1

samtools=1.15.1=h1170115_0

scipy=1.8.1=py38h1ee437e_0

seqtk=1.3=h7132678_4

setuptools=63.2.0=py38h578d9bd_0

six=1.16.0=pyh6c4a22f_0

sortedcontainers=2.4.0=pyhd8ed1ab_0

sqlite=3.39.1=h4ff8645_0

tar=1.34=ha1f6473_0

tensorboard=2.9.0=pyhd8ed1ab_0

tensorboard-data-server=0.6.0=py38h2b5fc30_2

tensorboard-plugin-wit=1.8.1=pyhd8ed1ab_0

tensorflow=2.2.0=gpu_py38hb782248_0

tensorflow-base=2.2.0=gpu_py38h83e3d50_0

tensorflow-estimator=2.6.0=py38h709712a_0

tensorflow-gpu=2.2.0=h0d30ee6_0

termcolor=1.1.0=pyhd8ed1ab_3

tk=8.6.12=h27826a3_0

typing-extensions=4.3.0=hd8ed1ab_0

typing_extensions=4.3.0=pyha770c72_0

urllib3=1.26.10=pyhd8ed1ab_0

werkzeug=2.1.2=pyhd8ed1ab_1

whatshap=1.4=py38h4a32c8e_0

wheel=0.37.1=pyhd8ed1ab_0

wrapt=1.14.1=py38h0a891b7_0

xopen=1.5.0=py38h578d9bd_0

xorg-fixesproto=5.0=h7f98852_1002

xorg-inputproto=2.3.2=h7f98852_1002

xorg-kbproto=1.0.7=h7f98852_1002

xorg-libice=1.0.10=h7f98852_0

xorg-libsm=1.2.3=hd9c2040_1000

xorg-libx11=1.7.2=h7f98852_0

xorg-libxau=1.0.9=h7f98852_0

xorg-libxdmcp=1.1.3=h7f98852_0

xorg-libxext=1.3.4=h7f98852_1

xorg-libxfixes=5.0.3=h7f98852_1004

xorg-libxi=1.7.10=h7f98852_0

xorg-libxrender=0.9.10=h7f98852_1003

xorg-libxtst=1.2.3=h7f98852_1002

xorg-recordproto=1.14.2=h7f98852_1002

xorg-renderproto=0.11.1=h7f98852_1002

xorg-xextproto=7.3.0=h7f98852_1002

xorg-xproto=7.0.31=h7f98852_1007

xz=5.2.5=h516909a_1

yarl=1.7.2=py38h0a891b7_2

zipp=3.8.0=pyhd8ed1ab_0

zlib=1.2.12=h166bdaf_2

zstd=1.5.2=h8a70e8d_2

Thank you!

Best regards, Eddie

Just to make sure, are expecting to run medaka on a GPU?

Only using CPU will be fine!

Try adding the CUDA_VISIBLE_DEVICES variable to your environment, then rerunning dragonflye. This should help us determine if its due to something on the gpu side of things or not.

export CUDA_VISIBLE_DEVICES=''

We just reinstalled the dragonflye env (v1.0.13) and test run with the following commands: conda activate dragonflye export CUDA_VISIBLE_DEVICES='' /mnt/data/Species-specific/CAUR$ dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --nanohq --model r941_min_sup_g507

But we seems to get the NotImplementedError. Please see the log blow:

(base) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ conda activate dragonflye (dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ export CUDA_VISIBLE_DEVICES='' (dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA: [dragonflye] Hello gilmansiu3 [dragonflye] You ran: /home/gilmansiu3/miniconda3/envs/dragonflye/bin/dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --nanohq --model r941_min_sup_g507 [dragonflye] This is dragonflye 1.0.13 [dragonflye] Written by Robert A Petit III [dragonflye] Homepage is https://github.com/rpetit3/dragonflye [dragonflye] Operating system is linux [dragonflye] Perl version is v5.32.1 [dragonflye] Machine has 20 CPU cores and 125.72 GB RAM [dragonflye] Verifying input model (--model): r941_min_sup_g507 [dragonflye] Model r941_min_sup_g507 verified! [dragonflye] Valid model provided, but number of Medaka rounds (--medaka) not given, assuming 1 round [dragonflye] Using any2fasta - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/any2fasta | any2fasta 0.4.2 [dragonflye] Using assembly-scan - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/assembly-scan | assembly-scan 0.4.1 [dragonflye] Using bwa - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/bwa | Version: 0.7.17-r1188 [dragonflye] Using fastp - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/fastp | fastp 0.23.2 [dragonflye] Using flye - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/flye | 2.9.1-b1780 [dragonflye] Using kmc - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/kmc | K-Mer Counter (KMC) ver. 3.2.1 (2022-01-04) [dragonflye] Using medaka - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/medaka | medaka 1.6.1 [dragonflye] Using miniasm - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/miniasm | 0.3-r179 [dragonflye] Using minimap2 - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/minimap2 | 2.24-r1122 [dragonflye] Using nanoq - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/nanoq | nanoq 0.9.0 [dragonflye] Using pigz - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/pigz | pigz 2.6 [dragonflye] Using pilon - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/pilon | Pilon version 1.24 Thu Jan 28 13:00:45 2021 -0500 [dragonflye] Using polypolish - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/polypolish | Polypolish v0.5.0 [dragonflye] Using porechop - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/porechop | 0.2.4 [dragonflye] Using racon - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/racon | 1.5.0 [dragonflye] Using rasusa - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/rasusa | rasusa 0.7.0 [dragonflye] Using raven - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/raven | 1.8.1 [dragonflye] Using samclip - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/samclip | samclip 0.4.0 [dragonflye] Using samtools - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/samtools | Version: 1.15.1 (using htslib 1.15.1) [dragonflye] Using seqtk - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/seqtk | Version: 1.3-r106 [dragonflye] Using tempdir: /tmp/xaLXJ629uH [dragonflye] Changing into folder: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1 [dragonflye] Collecting raw read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 \/mnt\/data\/Species-specific\/CAUR\/02_fastq\/220818_VRE1.fastq.gz 2>&1 1>/tmp/iq_46esfn0 | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Read stats: max_len = 45857 [dragonflye] Read stats: avg_len = 4267 [dragonflye] Read stats: total_bp = 432559422 [dragonflye] Read stats: min_len = 1000 [dragonflye] Using genome size 2800000 bp [dragonflye] Estimated sequencing depth: 154x [dragonflye] Filter reads based on length and/or quality [dragonflye] Running: nanoq --min-len 1000 --input /mnt/data/Species-specific/CAUR/02_fastq/220818_VRE1.fastq.gz --min-qual 0 2>&1 1> READS.filt.fq | sed 's/^/[nanoq] /' | tee -a dragonflye.log [dragonflye] Running: pigz -f -p 20 --fast READS.filt.fq 2>&1 | sed 's/^/[pigz] /' | tee -a dragonflye.log [dragonflye] No read depth reduction requested or necessary. [dragonflye] No read adapter trimming requested. [dragonflye] Running: ln -sf READS.filt.fq.gz READS.fq.gz 2>&1 | sed 's/^/[ln] /' | tee -a dragonflye.log [dragonflye] Collecting qc'd read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 READS.fq.gz 2>&1 1>/tmp/IvNeqTcrQQ | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Final Read stats: total_bp = 432559422 [dragonflye] Final Read stats: avg_len = 4267 [dragonflye] Final Read stats: max_len = 45857 [dragonflye] Final Read stats: min_len = 1000 [dragonflye] Average read length looks like 4267 bp [dragonflye] Assembling reads with 'flye' [dragonflye] Running: flye --nano-hq READS.fq.gz -g 2800000 -i 0 --threads 20 -o flye 2>&1 | sed 's/^/[flye] /' | tee -a dragonflye.log [flye] [2022-12-01 00:38:51] INFO: Starting Flye 2.9.1-b1780 [flye] [2022-12-01 00:38:51] INFO: >>>STAGE: configure [flye] [2022-12-01 00:38:51] INFO: Configuring run [flye] [2022-12-01 00:38:56] INFO: Total read length: 432559422 [flye] [2022-12-01 00:38:56] INFO: Input genome size: 2800000 [flye] [2022-12-01 00:38:56] INFO: Estimated coverage: 154 [flye] [2022-12-01 00:38:56] INFO: Reads N50/N90: 5835 / 2000 [flye] [2022-12-01 00:38:56] INFO: Minimum overlap set to 2000 [flye] [2022-12-01 00:38:56] INFO: >>>STAGE: assembly [flye] [2022-12-01 00:38:56] INFO: Assembling disjointigs [flye] [2022-12-01 00:38:56] INFO: Reading sequences [flye] [2022-12-01 00:39:00] INFO: Building minimizer index [flye] [2022-12-01 00:39:00] INFO: Pre-calculating index storage [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 00:39:05] INFO: Filling index [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 00:39:14] INFO: Extending reads [flye] [2022-12-01 00:39:57] INFO: Overlap-based coverage: 111 [flye] [2022-12-01 00:39:57] INFO: Median overlap divergence: 0.0539394 [flye] 0% 10% 90% 100% [flye] [2022-12-01 00:40:48] INFO: Assembled 6 disjointigs [flye] [2022-12-01 00:40:48] INFO: Generating sequence [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 00:40:49] INFO: Filtering contained disjointigs [flye] 0% 10% 30% 50% 60% 80% 100% [flye] [2022-12-01 00:40:49] INFO: Contained seqs: 0 [flye] [2022-12-01 00:40:50] INFO: >>>STAGE: consensus [flye] [2022-12-01 00:40:50] INFO: Running Minimap2 [flye] [2022-12-01 00:41:16] INFO: Computing consensus [flye] [2022-12-01 00:42:56] INFO: Alignment error rate: 0.068110 [flye] [2022-12-01 00:42:56] INFO: >>>STAGE: repeat [flye] [2022-12-01 00:42:56] INFO: Building and resolving repeat graph [flye] [2022-12-01 00:42:56] INFO: Parsing disjointigs [flye] [2022-12-01 00:42:56] INFO: Building repeat graph [flye] 0% 10% 30% 50% 60% 80% 100% [flye] [2022-12-01 00:42:57] INFO: Median overlap divergence: 0.00282849 [flye] [2022-12-01 00:42:57] INFO: Parsing reads [flye] [2022-12-01 00:43:01] INFO: Aligning reads to the graph [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 00:43:10] INFO: Aligned read sequence: 383410913 / 389252928 (0.984992) [flye] [2022-12-01 00:43:10] INFO: Median overlap divergence: 0.02589 [flye] [2022-12-01 00:43:10] INFO: Mean edge coverage: 122 [flye] [2022-12-01 00:43:10] INFO: Simplifying the graph [flye] [2022-12-01 00:43:11] INFO: >>>STAGE: contigger [flye] [2022-12-01 00:43:11] INFO: Generating contigs [flye] [2022-12-01 00:43:11] INFO: Reading sequences [flye] [2022-12-01 00:43:15] INFO: Generated 9 contigs [flye] [2022-12-01 00:43:15] INFO: Added 0 scaffold connections [flye] [2022-12-01 00:43:15] INFO: >>>STAGE: finalize [flye] [2022-12-01 00:43:15] INFO: Assembly statistics: [flye] [flye] Total length: 3170131 [flye] Fragments: 9 [flye] Fragments N50: 2791741 [flye] Largest frg: 2791741 [flye] Scaffolds: 0 [flye] Mean coverage: 120 [flye] [flye] [2022-12-01 00:43:15] INFO: Final assembly: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/assembly.fasta [dragonflye] Polishing with Racon (1 rounds) [dragonflye] Running: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz 2>&1 1> flye/polish/racon/1/aligments.paf | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0571.01] collected minimizers [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0632.47] sorted minimizers [polishing - racon (1 of 1)] [M::main::0.0632.47] loaded/built the index for 9 target sequence(s) [polishing - racon (1 of 1)] [M::mm_mapopt_update::0.0682.35] mid_occ = 26 [polishing - racon (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 9 [polishing - racon (1 of 1)] [M::mm_idx_stat::0.0722.28] distinct minimizers: 534333 (95.56% are singletons); average occurrences: 1.111; average spacing: 5.339; total length: 3170131 [polishing - racon (1 of 1)] [M::worker_pipeline::11.07511.97] mapped 101377 sequences [polishing - racon (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - racon (1 of 1)] [M::main] CMD: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz [polishing - racon (1 of 1)] [M::main] Real time: 11.080 sec; CPU: 132.624 sec; Peak RSS: 0.491 GB [dragonflye] Running: racon -t 20 READS.fq.gz flye/polish/racon/1/aligments.paf flye.fasta 2>&1 1> flye/polish/racon/1/consensus.fasta | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded target sequences 0.011683 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded sequences 4.829570 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded overlaps 0.086332 s [racon::Polisher::initialize] aligning overlaps [====================] 8.289367 s ] 0.501922 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] transformed data into windows 0.464560 s [racon::Polisher::polish] generating consensus [====================] 44.015550 s ] 1.292581 s [polishing - racon (1 of 1)] [racon::Polisher::] total = 57.762712 s [dragonflye] Polishing with Medaka (1 rounds) [dragonflye] Running: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log [polishing - medaka (1 of 1)] Checking program versions [polishing - medaka (1 of 1)] This is medaka 1.6.1 [polishing - medaka (1 of 1)] Program Version Required Pass
[polishing - medaka (1 of 1)] bcftools 1.15.1 1.11 True
[polishing - medaka (1 of 1)] bgzip 1.15.1 1.11 True
[polishing - medaka (1 of 1)] minimap2 2.24 2.11 True
[polishing - medaka (1 of 1)] samtools 1.15.1 1.11 True
[polishing - medaka (1 of 1)] tabix 1.15.1 1.11 True
[polishing - medaka (1 of 1)] Aligning basecalls to draft [polishing - medaka (1 of 1)] Creating fai index file /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.fai [polishing - medaka (1 of 1)] Creating mmi index file /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.0621.03] collected minimizers [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.0731.30] sorted minimizers [polishing - medaka (1 of 1)] [M::main::0.0891.24] loaded/built the index for 8 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 8 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.0931.23] distinct minimizers: 529378 (94.98% are singletons); average occurrences: 1.122; average spacing: 5.338; total length: 3170417 [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -I 16G -x map-ont -d /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta [polishing - medaka (1 of 1)] [M::main] Real time: 0.096 sec; CPU: 0.118 sec; Peak RSS: 0.033 GB [polishing - medaka (1 of 1)] [M::main::0.0181.03] loaded/built the index for 8 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_mapopt_update::0.0231.02] mid_occ = 27 [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 8 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.0271.02] distinct minimizers: 529378 (94.98% are singletons); average occurrences: 1.122; average spacing: 5.338; total length: 3170417 [polishing - medaka (1 of 1)] [M::worker_pipeline::21.10813.95] mapped 101377 sequences [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -x map-ont --secondary=no -L --MD -A 2 -B 4 -O 4,24 -E 2,1 -t 20 -a /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/READS.filt.fq.gz [polishing - medaka (1 of 1)] [M::main] Real time: 21.111 sec; CPU: 294.369 sec; Peak RSS: 2.029 GB [polishing - medaka (1 of 1)] [bam_sort_core] merging from 0 files and 20 in-memory blocks... [polishing - medaka (1 of 1)] Running medaka consensus [polishing - medaka (1 of 1)] [00:45:01 - Predict] Reducing threads to 2, anymore is a waste. [polishing - medaka (1 of 1)] 2022-12-01 00:45:01.202626: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected [polishing - medaka (1 of 1)] [00:45:01 - Predict] It looks like you are running medaka without a GPU and attempted to set a high number of threads. We have scaled this down to an optimal number. If you wish to improve performance please see https://nanoporetech.github.io/medaka/installation.html#improving-parallelism. [polishing - medaka (1 of 1)] [00:45:01 - Predict] Setting tensorflow inter/intra-op threads to 2/1. [polishing - medaka (1 of 1)] [00:45:01 - Predict] Processing region(s): contig_1:0-2793679 contig_2:0-235943 contig_3:0-7454 contig_5:0-45936 contig_6:0-8937 contig_7:0-32438 contig_8:0-34259 contig_9:0-11771 [polishing - medaka (1 of 1)] [00:45:01 - Predict] Using model: /home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/data/r941_min_sup_g507_model.tar.gz. [polishing - medaka (1 of 1)] [00:45:01 - Predict] Processing 10 long region(s) with batching. [polishing - medaka (1 of 1)] [00:45:01 - MdlStrTF] ModelStoreTF exception <class 'NotImplementedError'> [polishing - medaka (1 of 1)] Traceback (most recent call last): [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/bin/medaka", line 11, in [polishing - medaka (1 of 1)] sys.exit(main()) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/medaka.py", line 720, in main [polishing - medaka (1 of 1)] args.func(args) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/prediction.py", line 160, in predict [polishing - medaka (1 of 1)] model = model_store.load_model(time_steps=args.chunk_len) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/datastore.py", line 159, in load_model [polishing - medaka (1 of 1)] self.model = model_partial_function(time_steps=time_steps) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/models.py", line 147, in build_model [polishing - medaka (1 of 1)] model.add(Bidirectional(gru, input_shape=input_shape)) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 456, in _method_wrapper [polishing - medaka (1 of 1)] result = method(self, args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py", line 198, in add [polishing - medaka (1 of 1)] layer(x) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/wrappers.py", line 531, in call [polishing - medaka (1 of 1)] return super(Bidirectional, self).call(inputs, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in call [polishing - medaka (1 of 1)] outputs = call_fn(cast_inputs, args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/wrappers.py", line 644, in call [polishing - medaka (1 of 1)] y = self.forward_layer(forward_inputs, [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 654, in call [polishing - medaka (1 of 1)] return super(RNN, self).call(inputs, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in call [polishing - medaka (1 of 1)] outputs = call_fn(cast_inputs, *args, *kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent_v2.py", line 408, in call [polishing - medaka (1 of 1)] inputs, initialstate, = self._process_inputs(inputs, initial_state, None) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 848, in _process_inputs [polishing - medaka (1 of 1)] initial_state = self.get_initial_state(inputs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 636, in get_initial_state [polishing - medaka (1 of 1)] init_state = get_initial_state_fn( [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 1910, in get_initial_state [polishing - medaka (1 of 1)] return _generate_zero_filled_state_for_cell(self, inputs, batch_size, dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2926, in _generate_zero_filled_state_for_cell [polishing - medaka (1 of 1)] return _generate_zero_filled_state(batch_size, cell.state_size, dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2944, in _generate_zero_filled_state [polishing - medaka (1 of 1)] return create_zeros(state_size) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2939, in create_zeros [polishing - medaka (1 of 1)] return array_ops.zeros(init_state_size, dtype=dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2677, in wrapped [polishing - medaka (1 of 1)] tensor = fun(args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2721, in zeros [polishing - medaka (1 of 1)] output = _constant_if_small(zero, shape, dtype, name) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2662, in _constant_if_small [polishing - medaka (1 of 1)] if np.prod(shape) < 1000: [polishing - medaka (1 of 1)] File "<__array_function__ internals>", line 180, in prod [polishing - medaka (1 of 1)] File "/home/gilmansiu3/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 3045, in prod [polishing - medaka (1 of 1)] return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out, [polishing - medaka (1 of 1)] File "/home/gilmansiu3/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction [polishing - medaka (1 of 1)] return ufunc.reduce(obj, axis, dtype, out, passkwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 748, in array [polishing - medaka (1 of 1)] raise NotImplementedError("Cannot convert a symbolic Tensor ({}) to a numpy" [polishing - medaka (1 of 1)] NotImplementedError: Cannot convert a symbolic Tensor (bidirectional/forward_gru1/strided_slice:0) to a numpy array. [polishing - medaka (1 of 1)] Failed to run medaka consensus. [dragonflye] Error running command: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log

Hmmmm, does flye/polish/racon/1/consensus.fasta look like its not empty?

Yes, it is not empty.

Please see the attached file. (I renamed it with .txt extension because Github did not support .fasta format.) consensus.fasta.txt

Are you able to use Dragonflye with other samples?

Yes. I tried to perform assembly on E. coli this time. But it seems to get the same error. Please find the log below.

(dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ export CUDA_VISIBLE_DEVICES='' (dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ dragonflye --reads 02_fastq/Zymo_Escherichia_coli_ATCC_25922.fastq.gz --gsize 5M --outdir 04_dragonflye/Zymo_Escherichia_coli_ATCC_25922 --cpus 20 --nanohq --model r941_min_sup_g507 [dragonflye] Hello gilmansiu3 [dragonflye] You ran: /home/gilmansiu3/miniconda3/envs/dragonflye/bin/dragonflye --reads 02_fastq/Zymo_Escherichia_coli_ATCC_25922.fastq.gz --gsize 5M --outdir 04_dragonflye/Zymo_Escherichia_coli_ATCC_25922 --cpus 20 --nanohq --model r941_min_sup_g507 [dragonflye] This is dragonflye 1.0.13 [dragonflye] Written by Robert A Petit III [dragonflye] Homepage is https://github.com/rpetit3/dragonflye [dragonflye] Operating system is linux [dragonflye] Perl version is v5.32.1 [dragonflye] Machine has 20 CPU cores and 125.72 GB RAM [dragonflye] Verifying input model (--model): r941_min_sup_g507 [dragonflye] Model r941_min_sup_g507 verified! [dragonflye] Valid model provided, but number of Medaka rounds (--medaka) not given, assuming 1 round [dragonflye] Using any2fasta - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/any2fasta | any2fasta 0.4.2 [dragonflye] Using assembly-scan - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/assembly-scan | assembly-scan 0.4.1 [dragonflye] Using bwa - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/bwa | Version: 0.7.17-r1188 [dragonflye] Using fastp - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/fastp | fastp 0.23.2 [dragonflye] Using flye - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/flye | 2.9.1-b1780 [dragonflye] Using kmc - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/kmc | K-Mer Counter (KMC) ver. 3.2.1 (2022-01-04) [dragonflye] Using medaka - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/medaka | medaka 1.6.1 [dragonflye] Using miniasm - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/miniasm | 0.3-r179 [dragonflye] Using minimap2 - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/minimap2 | 2.24-r1122 [dragonflye] Using nanoq - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/nanoq | nanoq 0.9.0 [dragonflye] Using pigz - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/pigz | pigz 2.6 [dragonflye] Using pilon - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/pilon | Pilon version 1.24 Thu Jan 28 13:00:45 2021 -0500 [dragonflye] Using polypolish - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/polypolish | Polypolish v0.5.0 [dragonflye] Using porechop - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/porechop | 0.2.4 [dragonflye] Using racon - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/racon | 1.5.0 [dragonflye] Using rasusa - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/rasusa | rasusa 0.7.0 [dragonflye] Using raven - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/raven | 1.8.1 [dragonflye] Using samclip - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/samclip | samclip 0.4.0 [dragonflye] Using samtools - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/samtools | Version: 1.15.1 (using htslib 1.15.1) [dragonflye] Using seqtk - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/seqtk | Version: 1.3-r106 [dragonflye] Using tempdir: /tmp/s8fFyOxwj0 [dragonflye] Changing into folder: /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922 [dragonflye] Collecting raw read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 \/mnt\/data\/Species-specific\/CAUR\/02_fastq\/Zymo_Escherichia_coli_ATCC_25922.fastq.gz 2>&1 1>/tmp/cJKCrPwvUV | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Read stats: max_len = 79759 [dragonflye] Read stats: avg_len = 6288 [dragonflye] Read stats: total_bp = 1196562344 [dragonflye] Read stats: min_len = 1006 [dragonflye] Using genome size 5000000 bp [dragonflye] Estimated sequencing depth: 239x [dragonflye] Filter reads based on length and/or quality [dragonflye] Running: nanoq --min-len 1000 --input /mnt/data/Species-specific/CAUR/02_fastq/Zymo_Escherichia_coli_ATCC_25922.fastq.gz --min-qual 0 2>&1 1> READS.filt.fq | sed 's/^/[nanoq] /' | tee -a dragonflye.log [dragonflye] Running: pigz -f -p 20 --fast READS.filt.fq 2>&1 | sed 's/^/[pigz] /' | tee -a dragonflye.log [dragonflye] Subsampling reads from 239x to 150x [dragonflye] Running: rasusa -i READS.filt.fq.gz -c 150 -g 5000000 -s 42 2>&1 1> READS.sub.fq | sed 's/^/[rasusa] /' | tee -a dragonflye.log [rasusa] [2022-12-01][16:53:37][rasusa][INFO] Target number of bases to subsample to is: 750000000 [rasusa] [2022-12-01][16:53:37][rasusa][INFO] Gathering read lengths... [rasusa] [2022-12-01][16:53:46][rasusa][INFO] 190296 reads detected [rasusa] [2022-12-01][16:53:46][rasusa][INFO] Keeping 118951 reads [rasusa] [2022-12-01][16:53:56][rasusa][INFO] Actual coverage of kept reads is 150.00x [rasusa] [2022-12-01][16:53:56][rasusa][INFO] Done 🎉 [dragonflye] Running: pigz -f -p 20 --fast READS.sub.fq 2>&1 | sed 's/^/[pigz] /' | tee -a dragonflye.log [dragonflye] No read adapter trimming requested. [dragonflye] Running: ln -sf READS.sub.fq.gz READS.fq.gz 2>&1 | sed 's/^/[ln] /' | tee -a dragonflye.log [dragonflye] Collecting qc'd read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 READS.fq.gz 2>&1 1>/tmp/qdiQ_mghtw | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Final Read stats: avg_len = 6305 [dragonflye] Final Read stats: min_len = 1006 [dragonflye] Final Read stats: total_bp = 750001566 [dragonflye] Final Read stats: max_len = 79759 [dragonflye] Average read length looks like 6305 bp [dragonflye] Assembling reads with 'flye' [dragonflye] Running: flye --nano-hq READS.fq.gz -g 5000000 -i 0 --threads 20 -o flye 2>&1 | sed 's/^/[flye] /' | tee -a dragonflye.log [flye] [2022-12-01 16:54:08] INFO: Starting Flye 2.9.1-b1780 [flye] [2022-12-01 16:54:08] INFO: >>>STAGE: configure [flye] [2022-12-01 16:54:08] INFO: Configuring run [flye] [2022-12-01 16:54:17] INFO: Total read length: 750001566 [flye] [2022-12-01 16:54:17] INFO: Input genome size: 5000000 [flye] [2022-12-01 16:54:17] INFO: Estimated coverage: 150 [flye] [2022-12-01 16:54:17] INFO: Reads N50/N90: 9215 / 2848 [flye] [2022-12-01 16:54:17] INFO: Minimum overlap set to 3000 [flye] [2022-12-01 16:54:17] INFO: >>>STAGE: assembly [flye] [2022-12-01 16:54:17] INFO: Assembling disjointigs [flye] [2022-12-01 16:54:17] INFO: Reading sequences [flye] [2022-12-01 16:54:25] INFO: Building minimizer index [flye] [2022-12-01 16:54:25] INFO: Pre-calculating index storage [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 16:54:32] INFO: Filling index [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 16:54:51] INFO: Extending reads [flye] [2022-12-01 16:56:14] INFO: Overlap-based coverage: 120 [flye] [2022-12-01 16:56:14] INFO: Median overlap divergence: 0.0726324 [flye] 0% 90% 100% [flye] [2022-12-01 16:58:21] INFO: Assembled 5 disjointigs [flye] [2022-12-01 16:58:22] INFO: Generating sequence [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 16:58:24] INFO: Filtering contained disjointigs [flye] 0% 20% 40% 60% 80% 100% [flye] [2022-12-01 16:58:25] INFO: Contained seqs: 0 [flye] [2022-12-01 16:58:25] INFO: >>>STAGE: consensus [flye] [2022-12-01 16:58:25] INFO: Running Minimap2 [flye] [2022-12-01 16:59:07] INFO: Computing consensus [flye] [2022-12-01 17:00:16] INFO: Alignment error rate: 0.081555 [flye] [2022-12-01 17:00:16] INFO: >>>STAGE: repeat [flye] [2022-12-01 17:00:16] INFO: Building and resolving repeat graph [flye] [2022-12-01 17:00:16] INFO: Parsing disjointigs [flye] [2022-12-01 17:00:16] INFO: Building repeat graph [flye] 0% 20% 40% 60% 80% 100% [flye] [2022-12-01 17:00:18] INFO: Median overlap divergence: 0.0124352 [flye] [2022-12-01 17:00:18] INFO: Parsing reads [flye] [2022-12-01 17:00:25] INFO: Aligning reads to the graph [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 17:00:41] INFO: Aligned read sequence: 654011637 / 667036201 (0.980474) [flye] [2022-12-01 17:00:41] INFO: Median overlap divergence: 0.0368736 [flye] [2022-12-01 17:00:41] INFO: Mean edge coverage: 126 [flye] [2022-12-01 17:00:41] INFO: Simplifying the graph [flye] [2022-12-01 17:00:42] INFO: >>>STAGE: contigger [flye] [2022-12-01 17:00:42] INFO: Generating contigs [flye] [2022-12-01 17:00:42] INFO: Reading sequences [flye] [2022-12-01 17:00:49] INFO: Generated 5 contigs [flye] [2022-12-01 17:00:49] INFO: Added 0 scaffold connections [flye] [2022-12-01 17:00:50] INFO: >>>STAGE: finalize [flye] [2022-12-01 17:00:50] INFO: Assembly statistics: [flye] [flye] Total length: 5208142 [flye] Fragments: 5 [flye] Fragments N50: 5129207 [flye] Largest frg: 5129207 [flye] Scaffolds: 0 [flye] Mean coverage: 129 [flye] [flye] [2022-12-01 17:00:50] INFO: Final assembly: /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/assembly.fasta [dragonflye] Polishing with Racon (1 rounds) [dragonflye] Running: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz 2>&1 1> flye/polish/racon/1/aligments.paf | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0891.01] collected minimizers [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0992.68] sorted minimizers [polishing - racon (1 of 1)] [M::main::0.0992.68] loaded/built the index for 5 target sequence(s) [polishing - racon (1 of 1)] [M::mm_mapopt_update::0.1092.53] mid_occ = 10 [polishing - racon (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 5 [polishing - racon (1 of 1)] [M::mm_idx_stat::0.1172.43] distinct minimizers: 942970 (97.58% are singletons); average occurrences: 1.032; average spacing: 5.351; total length: 5208142 [polishing - racon (1 of 1)] [M::worker_pipeline::9.0619.33] mapped 78175 sequences [polishing - racon (1 of 1)] [M::worker_pipeline::11.40711.26] mapped 40776 sequences [polishing - racon (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - racon (1 of 1)] [M::main] CMD: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz [polishing - racon (1 of 1)] [M::main] Real time: 11.412 sec; CPU: 128.446 sec; Peak RSS: 0.807 GB [dragonflye] Running: racon -t 20 READS.fq.gz flye/polish/racon/1/aligments.paf flye.fasta 2>&1 1> flye/polish/racon/1/consensus.fasta | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded target sequences 0.021195 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded sequences 8.468119 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded overlaps 0.106399 s [racon::Polisher::initialize] aligning overlaps [====================] 19.257848 s ] 1.160865 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] transformed data into windows 0.800375 s [racon::Polisher::polish] generating consensus [====================] 83.666797 s ] 4.421268 s [polishing - racon (1 of 1)] [racon::Polisher::] total = 112.396382 s [dragonflye] Polishing with Medaka (1 rounds) [dragonflye] Running: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log [polishing - medaka (1 of 1)] Checking program versions [polishing - medaka (1 of 1)] This is medaka 1.6.1 [polishing - medaka (1 of 1)] Program Version Required Pass
[polishing - medaka (1 of 1)] bcftools 1.15.1 1.11 True
[polishing - medaka (1 of 1)] bgzip 1.15.1 1.11 True
[polishing - medaka (1 of 1)] minimap2 2.24 2.11 True
[polishing - medaka (1 of 1)] samtools 1.15.1 1.11 True
[polishing - medaka (1 of 1)] tabix 1.15.1 1.11 True
[polishing - medaka (1 of 1)] Aligning basecalls to draft [polishing - medaka (1 of 1)] Creating fai index file /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/polish/racon/1/consensus.fasta.fai [polishing - medaka (1 of 1)] Creating mmi index file /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/polish/racon/1/consensus.fasta.map-ont.mmi [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.1411.02] collected minimizers [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.1621.27] sorted minimizers [polishing - medaka (1 of 1)] [M::main::0.1901.23] loaded/built the index for 5 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 5 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.1981.22] distinct minimizers: 939300 (97.39% are singletons); average occurrences: 1.037; average spacing: 5.350; total length: 5209708 [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -I 16G -x map-ont -d /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/polish/racon/1/consensus.fasta [polishing - medaka (1 of 1)] [M::main] Real time: 0.201 sec; CPU: 0.244 sec; Peak RSS: 0.053 GB [polishing - medaka (1 of 1)] [M::main::0.0351.01] loaded/built the index for 5 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_mapopt_update::0.0441.01] mid_occ = 10 [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 5 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.0521.01] distinct minimizers: 939300 (97.39% are singletons); average occurrences: 1.037; average spacing: 5.350; total length: 5209708 [polishing - medaka (1 of 1)] [M::worker_pipeline::26.37714.81] mapped 78175 sequences [polishing - medaka (1 of 1)] [M::worker_pipeline::27.70514.15] mapped 40776 sequences [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -x map-ont --secondary=no -L --MD -A 2 -B 4 -O 4,24 -E 2,1 -t 20 -a /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/READS.sub.fq.gz [polishing - medaka (1 of 1)] [M::main] Real time: 27.743 sec; CPU: 392.046 sec; Peak RSS: 2.452 GB [polishing - medaka (1 of 1)] [bam_sort_core] merging from 0 files and 20 in-memory blocks... [polishing - medaka (1 of 1)] Running medaka consensus [polishing - medaka (1 of 1)] [17:03:44 - Predict] Reducing threads to 2, anymore is a waste. [polishing - medaka (1 of 1)] 2022-12-01 17:03:44.334565: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected [polishing - medaka (1 of 1)] [17:03:44 - Predict] It looks like you are running medaka without a GPU and attempted to set a high number of threads. We have scaled this down to an optimal number. If you wish to improve performance please see https://nanoporetech.github.io/medaka/installation.html#improving-parallelism. [polishing - medaka (1 of 1)] [17:03:44 - Predict] Setting tensorflow inter/intra-op threads to 2/1. [polishing - medaka (1 of 1)] [17:03:44 - Predict] Processing region(s): contig_1:0-5131525 contig_2:0-2989 contig_3:0-5557 contig_4:0-45470 contig_5:0-24167 [polishing - medaka (1 of 1)] [17:03:44 - Predict] Using model: /home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/data/r941_min_sup_g507_model.tar.gz. [polishing - medaka (1 of 1)] [17:03:44 - Predict] Processing 10 long region(s) with batching. [polishing - medaka (1 of 1)] [17:03:44 - MdlStrTF] ModelStoreTF exception <class 'NotImplementedError'> [polishing - medaka (1 of 1)] Traceback (most recent call last): [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/bin/medaka", line 11, in [polishing - medaka (1 of 1)] sys.exit(main()) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/medaka.py", line 720, in main [polishing - medaka (1 of 1)] args.func(args) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/prediction.py", line 160, in predict [polishing - medaka (1 of 1)] model = model_store.load_model(time_steps=args.chunk_len) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/datastore.py", line 159, in load_model [polishing - medaka (1 of 1)] self.model = model_partial_function(time_steps=time_steps) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/models.py", line 147, in build_model [polishing - medaka (1 of 1)] model.add(Bidirectional(gru, input_shape=input_shape)) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 456, in _method_wrapper [polishing - medaka (1 of 1)] result = method(self, args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py", line 198, in add [polishing - medaka (1 of 1)] layer(x) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/wrappers.py", line 531, in call [polishing - medaka (1 of 1)] return super(Bidirectional, self).call(inputs, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in call [polishing - medaka (1 of 1)] outputs = call_fn(cast_inputs, args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/wrappers.py", line 644, in call [polishing - medaka (1 of 1)] y = self.forward_layer(forward_inputs, [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 654, in call [polishing - medaka (1 of 1)] return super(RNN, self).call(inputs, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in call [polishing - medaka (1 of 1)] outputs = call_fn(cast_inputs, *args, *kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent_v2.py", line 408, in call [polishing - medaka (1 of 1)] inputs, initialstate, = self._process_inputs(inputs, initial_state, None) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 848, in _process_inputs [polishing - medaka (1 of 1)] initial_state = self.get_initial_state(inputs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 636, in get_initial_state [polishing - medaka (1 of 1)] init_state = get_initial_state_fn( [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 1910, in get_initial_state [polishing - medaka (1 of 1)] return _generate_zero_filled_state_for_cell(self, inputs, batch_size, dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2926, in _generate_zero_filled_state_for_cell [polishing - medaka (1 of 1)] return _generate_zero_filled_state(batch_size, cell.state_size, dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2944, in _generate_zero_filled_state [polishing - medaka (1 of 1)] return create_zeros(state_size) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2939, in create_zeros [polishing - medaka (1 of 1)] return array_ops.zeros(init_state_size, dtype=dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2677, in wrapped [polishing - medaka (1 of 1)] tensor = fun(args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2721, in zeros [polishing - medaka (1 of 1)] output = _constant_if_small(zero, shape, dtype, name) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2662, in _constant_if_small [polishing - medaka (1 of 1)] if np.prod(shape) < 1000: [polishing - medaka (1 of 1)] File "<__array_function__ internals>", line 180, in prod [polishing - medaka (1 of 1)] File "/home/gilmansiu3/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 3045, in prod [polishing - medaka (1 of 1)] return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out, [polishing - medaka (1 of 1)] File "/home/gilmansiu3/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction [polishing - medaka (1 of 1)] return ufunc.reduce(obj, axis, dtype, out, passkwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 748, in array [polishing - medaka (1 of 1)] raise NotImplementedError("Cannot convert a symbolic Tensor ({}) to a numpy" [polishing - medaka (1 of 1)] NotImplementedError: Cannot convert a symbolic Tensor (bidirectional/forward_gru1/strided_slice:0) to a numpy array. [polishing - medaka (1 of 1)] Failed to run medaka consensus. [dragonflye] Error running command: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log (dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$

I wonder what would happen if you used a different version of medaka.

Maybe try:

mamba create -n test-dragonflye -c conda-forge -c bioconda dragonflye 'medaka>1.6.1'

I just tried to create the test-dragonflye env with medaka>1.6.1, but seems it could only get dragonflye=1.0.7.

(base) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ mamba create -n test-dragonflye -c conda-forge -c bioconda dragonflye 'medaka>1.6.1'
              __    __    __    __
             /  \  /  \  /  \  /  \
            /    \/    \/    \/    \
███████████████/ /██/ /██/ /██/ /████████████████████████ / / \ / \ / \ / \ _ / / _/ _/ _/ \ o _, / _/ ___/ ` |/ ███╗ ███╗ █████╗ ███╗ ███╗██████╗ █████╗ ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗ ██╔████╔██║███████║██╔████╔██║██████╔╝███████║ ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║ ██║ ╚═╝ ██║██║ ██║██║ ╚═╝ ██║██████╔╝██║ ██║ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝
    mamba (0.22.1) supported by @QuantStack

    GitHub:  https://github.com/mamba-org/mamba
    Twitter: https://twitter.com/QuantStack
█████████████████████████████████████████████████████████████

Looking for: ['dragonflye', "medaka[version='>1.6.1']"]

conda-forge/linux-64 Using cache conda-forge/noarch Using cache bioconda/linux-64 Using cache bioconda/noarch Using cache r/linux-64 Using cache r/noarch Using cache pkgs/r/linux-64 No change pkgs/r/noarch No change pkgs/main/linux-64 No change pkgs/main/noarch No change cruizperez/linux-64 No change liulab-dfci/noarch No change cruizperez/noarch No change liulab-dfci/linux-64 No change Transaction

Prefix: /home/gilmansiu3/miniconda3/envs/test-dragonflye

Updating specs:

dragonflye

medaka[version='>1.6.1']

Package Version Build Channel Size ──────────────────────────────────────────────────────────────────────────────────────────────────── Install: ────────────────────────────────────────────────────────────────────────────────────────────────────

_libgcc_mutex 0.1 conda_forge conda-forge/linux-64 Cached

_openmp_mutex 4.5 2_gnu conda-forge/linux-64 Cached

abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge/linux-64 Cached

absl-py 1.3.0 pyhd8ed1ab_0 conda-forge/noarch Cached

aiohttp 3.8.3 py38h0a891b7_1 conda-forge/linux-64 Cached

aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge/noarch Cached

any2fasta 0.4.2 hdfd78af_3 bioconda/noarch Cached

assembly-scan 0.4.1 pyhdfd78af_0 bioconda/noarch Cached

astunparse 1.6.3 pyhd8ed1ab_0 conda-forge/noarch Cached

async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge/noarch Cached

attrs 22.1.0 pyh71513ae_1 conda-forge/noarch Cached

bcftools 1.16 hfe4b78e_1 bioconda/linux-64 Cached

biopython 1.80 py38h0a891b7_0 conda-forge/linux-64 Cached

blinker 1.5 pyhd8ed1ab_0 conda-forge/noarch Cached

brotlipy 0.7.0 py38h0a891b7_1005 conda-forge/linux-64 Cached

bzip2 1.0.8 h7f98852_4 conda-forge/linux-64 Cached

c-ares 1.18.1 h7f98852_0 conda-forge/linux-64 Cached

ca-certificates 2022.9.24 ha878542_0 conda-forge/linux-64 Cached

cached-property 1.5.2 hd8ed1ab_1 conda-forge/noarch Cached

cached_property 1.5.2 pyha770c72_1 conda-forge/noarch Cached

cachetools 4.2.4 pyhd8ed1ab_0 conda-forge/noarch Cached

certifi 2022.9.24 pyhd8ed1ab_0 conda-forge/noarch Cached

cffi 1.15.1 py38h4a40e3a_2 conda-forge/linux-64 Cached

charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge/noarch Cached

click 8.1.3 unix_pyhd8ed1ab_2 conda-forge/noarch Cached

cryptography 38.0.4 py38h2b5fc30_0 conda-forge/linux-64 Cached

dragonflye 1.0.7 hdfd78af_0 bioconda/noarch Cached

flye 2.9.1 py38hf4f3596_0 bioconda/linux-64 Cached

frozenlist 1.3.3 py38h0a891b7_0 conda-forge/linux-64 Cached

gast 0.4.0 pyh9f0ad1d_0 conda-forge/noarch Cached

giflib 5.2.1 h36c2ea0_2 conda-forge/linux-64 Cached

google-auth 1.35.0 pyh6c4a22f_0 conda-forge/noarch Cached

google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge/noarch Cached

google-pasta 0.2.0 pyh8c360ce_0 conda-forge/noarch Cached

grpc-cpp 1.43.2 h9e046d8_3 conda-forge/linux-64 Cached

grpcio 1.43.0 py38hdd6454d_0 conda-forge/linux-64 Cached

gsl 2.7 he838d99_0 conda-forge/linux-64 Cached

h5py 3.7.0 nompi_py38h045baee_101 conda-forge/linux-64 Cached

hdf5 1.12.2 nompi_h2386368_100 conda-forge/linux-64 Cached

htslib 1.16 h6bc39ce_0 bioconda/linux-64 Cached

icu 70.1 h27087fc_0 conda-forge/linux-64 Cached

idna 3.4 pyhd8ed1ab_0 conda-forge/noarch Cached

importlib-metadata 5.1.0 pyha770c72_0 conda-forge/noarch Cached

intervaltree 3.0.2 py_0 conda-forge/noarch Cached

jpeg 9e h166bdaf_2 conda-forge/linux-64 Cached

k8 0.2.5 hd03093a_2 bioconda/linux-64 Cached

keras 2.7.0 pyhd8ed1ab_0 conda-forge/noarch Cached

keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge/noarch Cached

keyutils 1.6.1 h166bdaf_0 conda-forge/linux-64 Cached

kmc 3.2.1 hf1761c0_2 bioconda/linux-64 Cached

krb5 1.19.3 h3790be6_0 conda-forge/linux-64 Cached

ld_impl_linux-64 2.39 hcc3a1bd_1 conda-forge/linux-64 Cached

libblas 3.9.0 16_linux64_openblas conda-forge/linux-64 Cached

libcblas 3.9.0 16_linux64_openblas conda-forge/linux-64 Cached

libcurl 7.86.0 h7bff187_1 conda-forge/linux-64 Cached

libdeflate 1.13 h166bdaf_0 conda-forge/linux-64 Cached

libedit 3.1.20191231 he28a2e2_2 conda-forge/linux-64 Cached

libev 4.33 h516909a_1 conda-forge/linux-64 Cached

libffi 3.4.2 h7f98852_5 conda-forge/linux-64 Cached

libgcc-ng 12.2.0 h65d4601_19 conda-forge/linux-64 Cached

libgfortran-ng 12.2.0 h69a702a_19 conda-forge/linux-64 Cached

libgfortran5 12.2.0 h337968e_19 conda-forge/linux-64 Cached

libgomp 12.2.0 h65d4601_19 conda-forge/linux-64 Cached

libiconv 1.17 h166bdaf_0 conda-forge/linux-64 Cached

liblapack 3.9.0 16_linux64_openblas conda-forge/linux-64 Cached

libnghttp2 1.47.0 hdcd2b5c_1 conda-forge/linux-64 Cached

libnsl 2.0.0 h7f98852_0 conda-forge/linux-64 Cached

libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge/linux-64 Cached

libpng 1.6.39 h753d276_0 conda-forge/linux-64 Cached

libprotobuf 3.19.4 h780b84a_0 conda-forge/linux-64 Cached

libsqlite 3.40.0 h753d276_0 conda-forge/linux-64 Cached

libssh2 1.10.0 haa6b8db_3 conda-forge/linux-64 Cached

libstdcxx-ng 12.2.0 h46fd767_19 conda-forge/linux-64 Cached

libuuid 2.32.1 h7f98852_1000 conda-forge/linux-64 Cached

libzlib 1.2.13 h166bdaf_4 conda-forge/linux-64 Cached

mappy 2.24 py38h4c6a040_1 bioconda/linux-64 Cached

markdown 3.4.1 pyhd8ed1ab_0 conda-forge/noarch Cached

markupsafe 2.1.1 py38h0a891b7_2 conda-forge/linux-64 Cached

medaka 1.7.2 py38h6239ad4_1 bioconda/linux-64 Cached

miniasm 0.3_r179 h7132678_3 bioconda/linux-64 Cached

minimap2 2.24 h7132678_1 bioconda/linux-64 Cached

multidict 6.0.2 py38h0a891b7_2 conda-forge/linux-64 Cached

nanoq 0.9.0 hec16e2b_1 bioconda/linux-64 Cached

ncurses 6.3 h27087fc_1 conda-forge/linux-64 Cached

numpy 1.19.5 py38h8246c76_3 conda-forge/linux-64 Cached

oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge/noarch Cached

ont-fast5-api 4.1.0 pyhdfd78af_0 bioconda/noarch Cached

openssl 1.1.1s h0b41bf4_1 conda-forge/linux-64 Cached

opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge/noarch Cached

packaging 21.3 pyhd8ed1ab_0 conda-forge/noarch Cached

parasail-python 1.3.3 py38h3b68952_0 bioconda/linux-64 Cached

perl 5.32.1 2_h7f98852_perl5 conda-forge/linux-64 Cached

perl-file-spec 3.48_01 pl5321hdfd78af_2 bioconda/noarch Cached

perl-findbin 1.51 pl5321hdfd78af_3 bioconda/noarch Cached

pigz 2.6 h27826a3_0 conda-forge/linux-64 Cached

pip 22.3.1 pyhd8ed1ab_0 conda-forge/noarch Cached

progressbar33 2.4 py_0 conda-forge/noarch Cached

protobuf 3.19.4 py38h709712a_0 conda-forge/linux-64 Cached

pyasn1 0.4.8 py_0 conda-forge/noarch Cached

pyasn1-modules 0.2.7 py_0 conda-forge/noarch Cached

pycparser 2.21 pyhd8ed1ab_0 conda-forge/noarch Cached

pyjwt 2.6.0 pyhd8ed1ab_0 conda-forge/noarch Cached

pyopenssl 22.1.0 pyhd8ed1ab_0 conda-forge/noarch Cached

pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge/noarch Cached

pysam 0.19.1 py38h1c8baaf_1 bioconda/linux-64 Cached

pysocks 1.7.1 pyha2e5f31_6 conda-forge/noarch Cached

pyspoa 0.0.3 py38h8ded8fe_3 bioconda/linux-64 Cached

python 3.8.15 h257c98d_0_cpython conda-forge/linux-64 Cached

python-edlib 1.3.9 py38h4a32c8e_1 bioconda/linux-64 Cached

python-flatbuffers 2.0 pyhd8ed1ab_0 conda-forge/noarch Cached

python_abi 3.8 3_cp38 conda-forge/linux-64 Cached

pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge/noarch Cached

racon 1.5.0 h7ff8a90_0 bioconda/linux-64 Cached

rasusa 0.7.0 hec16e2b_1 bioconda/linux-64 Cached

raven-assembler 1.8.1 h5b5514e_1 bioconda/linux-64 Cached

re2 2022.02.01 h9c3ff4c_0 conda-forge/linux-64 Cached

readline 8.1.2 h0f457ee_0 conda-forge/linux-64 Cached

requests 2.28.1 pyhd8ed1ab_1 conda-forge/noarch Cached

requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge/noarch Cached

rsa 4.9 pyhd8ed1ab_0 conda-forge/noarch Cached

samtools 1.16.1 h6899075_1 bioconda/linux-64 Cached

scipy 1.9.1 py38hea3f02b_0 conda-forge/linux-64 Cached

seqtk 1.3 h7132678_4 bioconda/linux-64 Cached

setuptools 65.5.1 pyhd8ed1ab_0 conda-forge/noarch Cached

six 1.16.0 pyh6c4a22f_0 conda-forge/noarch Cached

snappy 1.1.9 hbd366e4_2 conda-forge/linux-64 Cached

sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge/noarch Cached

sqlite 3.40.0 h4ff8645_0 conda-forge/linux-64 Cached

tar 1.34 hb2e2bae_1 conda-forge/linux-64 Cached

tensorboard 2.6.0 pyhd8ed1ab_1 conda-forge/noarch Cached

tensorboard-data-server 0.6.1 py38h2b5fc30_4 conda-forge/linux-64 Cached

tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge/noarch Cached

tensorflow 2.7.1 cpu_py38h66f0ec1_0 conda-forge/linux-64 Cached

tensorflow-base 2.7.1 cpu_py38ha28dbe6_0 conda-forge/linux-64 Cached

tensorflow-estimator 2.7.1 cpu_py38h4e23bc6_0 conda-forge/linux-64 Cached

termcolor 2.1.1 pyhd8ed1ab_0 conda-forge/noarch Cached

tk 8.6.12 h27826a3_0 conda-forge/linux-64 Cached

typing-extensions 4.4.0 hd8ed1ab_0 conda-forge/noarch Cached

typing_extensions 4.4.0 pyha770c72_0 conda-forge/noarch Cached

urllib3 1.26.13 pyhd8ed1ab_0 conda-forge/noarch Cached

werkzeug 2.2.2 pyhd8ed1ab_0 conda-forge/noarch Cached

wheel 0.38.4 pyhd8ed1ab_0 conda-forge/noarch Cached

wrapt 1.14.1 py38h0a891b7_1 conda-forge/linux-64 Cached

xz 5.2.6 h166bdaf_0 conda-forge/linux-64 Cached

yarl 1.8.1 py38h0a891b7_0 conda-forge/linux-64 Cached

zipp 3.11.0 pyhd8ed1ab_0 conda-forge/noarch Cached

zlib 1.2.13 h166bdaf_4 conda-forge/linux-64 Cached

zstd 1.5.2 h6239696_4 conda-forge/linux-64 Cached

Summary:

Install: 148 packages

Total download: 0 B

────────────────────────────────────────────────────────────────────────────────────────────────────

Confirm changes: [Y/n] y Preparing transaction: done Verifying transaction: done Executing transaction: done

To activate this environment, use
 $ mamba activate test-dragonflye
To deactivate an active environment, use
 $ mamba deactivate
(base) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$

Also, I just tried to use the test-dragonflye env with dragonflye=1.0.7 & medaka=1.7.2 and it works. But it could not support the option --nanohq as it is using an older version of dragonflye.

(base) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ conda activate test-dragonflye (test-dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --model r941_min_sup_g507 [dragonflye] Hello gilmansiu3 [dragonflye] You ran: /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --model r941_min_sup_g507 [dragonflye] This is dragonflye 1.0.7 [dragonflye] Written by Robert A Petit III [dragonflye] Homepage is https://github.com/rpetit3/dragonflye [dragonflye] Operating system is linux [dragonflye] Perl version is v5.32.1 [dragonflye] Machine has 20 CPU cores and 125.72 GB RAM [dragonflye] Verifying input model (--model): r941_min_sup_g507 [dragonflye] Model r941_min_sup_g507 verified! [dragonflye] Valid model provided, but number of Medaka rounds (--medaka) not given, assuming 1 round [dragonflye] Using any2fasta - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/any2fasta | any2fasta 0.4.2 [dragonflye] Using assembly-scan - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/assembly-scan | assembly-scan 0.4.1 [dragonflye] Using flye - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/flye | 2.9.1-b1780 [dragonflye] Using kmc - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/kmc | K-Mer Counter (KMC) ver. 3.2.1 (2022-01-04) [dragonflye] Using medaka - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/medaka | medaka 1.7.2 [dragonflye] Using miniasm - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/miniasm | 0.3-r179 [dragonflye] Using minimap2 - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/minimap2 | 2.24-r1122 [dragonflye] Using nanoq - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/nanoq | nanoq 0.9.0 [dragonflye] Using pigz - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/pigz | pigz 2.6 [dragonflye] Using racon - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/racon | 1.5.0 [dragonflye] Using rasusa - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/rasusa | rasusa 0.7.0 [dragonflye] Using raven - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/raven | 1.8.1 [dragonflye] Using seqtk - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/seqtk | Version: 1.3-r106 [dragonflye] Using tempdir: /tmp/vmMKaDqOh4 [dragonflye] Changing into folder: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1 [dragonflye] Collecting raw read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 \/mnt\/data\/Species-specific\/CAUR\/02_fastq\/220818_VRE1.fastq.gz 2>&1 1>/tmp/RGfzcT9ipW | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Read stats: max_len = 45857 [dragonflye] Read stats: total_bp = 432559422 [dragonflye] Read stats: avg_len = 4267 [dragonflye] Read stats: min_len = 1000 [dragonflye] Using genome size 2800000 bp [dragonflye] Estimated sequencing depth: 154x [dragonflye] Filter reads based on length and/or quality [dragonflye] Running: nanoq --min-len 1000 --input /mnt/data/Species-specific/CAUR/02_fastq/220818_VRE1.fastq.gz --min-qual 0 2>&1 1> READS.filt.fq | sed 's/^/[nanoq] /' | tee -a dragonflye.log [dragonflye] Running: pigz -f -p 20 --fast READS.filt.fq 2>&1 | sed 's/^/[pigz] /' | tee -a dragonflye.log [dragonflye] No read depth reduction requested or necessary. [dragonflye] Running: ln -sf READS.filt.fq.gz READS.fq.gz 2>&1 | sed 's/^/[ln] /' | tee -a dragonflye.log [dragonflye] Average read length looks like 4267 bp [dragonflye] Assembling reads with 'flye' [dragonflye] Running: flye --nano-raw READS.fq.gz -g 2800000 -i 0 --threads 20 -o flye 2>&1 | sed 's/^/[flye] /' | tee -a dragonflye.log [flye] [2022-12-05 11:47:23] INFO: Starting Flye 2.9.1-b1780 [flye] [2022-12-05 11:47:23] INFO: >>>STAGE: configure [flye] [2022-12-05 11:47:23] INFO: Configuring run [flye] [2022-12-05 11:47:29] INFO: Total read length: 432559422 [flye] [2022-12-05 11:47:29] INFO: Input genome size: 2800000 [flye] [2022-12-05 11:47:29] INFO: Estimated coverage: 154 [flye] [2022-12-05 11:47:29] INFO: Reads N50/N90: 5835 / 2000 [flye] [2022-12-05 11:47:29] INFO: Minimum overlap set to 2000 [flye] [2022-12-05 11:47:29] INFO: >>>STAGE: assembly [flye] [2022-12-05 11:47:29] INFO: Assembling disjointigs [flye] [2022-12-05 11:47:29] INFO: Reading sequences [flye] [2022-12-05 11:47:35] INFO: Counting k-mers: [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-05 11:48:22] INFO: Filling index table (1/2) [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-05 11:48:35] INFO: Filling index table (2/2) [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-05 11:48:58] INFO: Extending reads [flye] [2022-12-05 11:49:58] INFO: Overlap-based coverage: 115 [flye] [2022-12-05 11:49:58] INFO: Median overlap divergence: 0.0611082 [flye] 0% 10% 90% 100% [flye] [2022-12-05 11:51:10] INFO: Assembled 6 disjointigs [flye] [2022-12-05 11:51:10] INFO: Generating sequence [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-05 11:51:12] INFO: Filtering contained disjointigs [flye] 0% 10% 30% 50% 60% 80% 100% [flye] [2022-12-05 11:51:13] INFO: Contained seqs: 0 [flye] [2022-12-05 11:51:13] INFO: >>>STAGE: consensus [flye] [2022-12-05 11:51:13] INFO: Running Minimap2 [flye] [2022-12-05 11:51:40] INFO: Computing consensus [flye] [2022-12-05 11:53:26] INFO: Alignment error rate: 0.072399 [flye] [2022-12-05 11:53:26] INFO: >>>STAGE: repeat [flye] [2022-12-05 11:53:26] INFO: Building and resolving repeat graph [flye] [2022-12-05 11:53:26] INFO: Parsing disjointigs [flye] [2022-12-05 11:53:26] INFO: Building repeat graph [flye] 0% 10% 30% 50% 60% 80% 100% [flye] [2022-12-05 11:53:29] INFO: Median overlap divergence: 0.00331187 [flye] [2022-12-05 11:53:29] INFO: Parsing reads [flye] [2022-12-05 11:53:34] INFO: Aligning reads to the graph [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-05 11:53:50] INFO: Aligned read sequence: 385741125 / 389252928 (0.990978) [flye] [2022-12-05 11:53:50] INFO: Median overlap divergence: 0.016065 [flye] [2022-12-05 11:53:50] INFO: Mean edge coverage: 123 [flye] [2022-12-05 11:53:50] INFO: Simplifying the graph [flye] [2022-12-05 11:53:50] INFO: >>>STAGE: contigger [flye] [2022-12-05 11:53:50] INFO: Generating contigs [flye] [2022-12-05 11:53:50] INFO: Reading sequences [flye] [2022-12-05 11:53:55] INFO: Generated 6 contigs [flye] [2022-12-05 11:53:55] INFO: Added 0 scaffold connections [flye] [2022-12-05 11:53:55] INFO: >>>STAGE: finalize [flye] [2022-12-05 11:53:55] INFO: Assembly statistics: [flye] [flye] Total length: 3121116 [flye] Fragments: 6 [flye] Fragments N50: 2791405 [flye] Largest frg: 2791405 [flye] Scaffolds: 0 [flye] Mean coverage: 122 [flye] [flye] [2022-12-05 11:53:55] INFO: Final assembly: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/assembly.fasta [dragonflye] Polishing with Racon (1 rounds) [dragonflye] Running: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz 2>&1 1> flye/polish/racon/1/aligments.paf | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0571.01] collected minimizers [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0632.43] sorted minimizers [polishing - racon (1 of 1)] [M::main::0.0632.43] loaded/built the index for 6 target sequence(s) [polishing - racon (1 of 1)] [M::mm_mapopt_update::0.0692.31] mid_occ = 25 [polishing - racon (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 6 [polishing - racon (1 of 1)] [M::mm_idx_stat::0.0742.22] distinct minimizers: 531318 (96.61% are singletons); average occurrences: 1.100; average spacing: 5.339; total length: 3121116 [polishing - racon (1 of 1)] [M::worker_pipeline::11.12011.61] mapped 101377 sequences [polishing - racon (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - racon (1 of 1)] [M::main] CMD: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz [polishing - racon (1 of 1)] [M::main] Real time: 11.125 sec; CPU: 129.068 sec; Peak RSS: 0.491 GB [dragonflye] Running: racon -t 20 READS.fq.gz flye/polish/racon/1/aligments.paf flye.fasta 2>&1 1> flye/polish/racon/1/consensus.fasta | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded target sequences 0.011601 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded sequences 4.978672 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded overlaps 0.090052 s [racon::Polisher::initialize] aligning overlaps [====================] 8.640065 s ] 0.737275 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] transformed data into windows 0.508076 s [racon::Polisher::polish] generating consensus [====================] 47.499805 s ] 2.754352 s [polishing - racon (1 of 1)] [racon::Polisher::] total = 61.797770 s [dragonflye] Polishing with Medaka (1 rounds) [dragonflye] Running: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log [polishing - medaka (1 of 1)] Checking program versions [polishing - medaka (1 of 1)] This is medaka 1.7.2 [polishing - medaka (1 of 1)] Program Version Required Pass
[polishing - medaka (1 of 1)] bcftools 1.16 1.11 True
[polishing - medaka (1 of 1)] bgzip 1.16 1.11 True
[polishing - medaka (1 of 1)] minimap2 2.24 2.11 True
[polishing - medaka (1 of 1)] samtools 1.16.1 1.11 True
[polishing - medaka (1 of 1)] tabix 1.16 1.11 True
[polishing - medaka (1 of 1)] Aligning basecalls to draft [polishing - medaka (1 of 1)] Creating fai index file /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.fai [polishing - medaka (1 of 1)] Creating mmi index file /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.0941.03] collected minimizers [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.1041.21] sorted minimizers [polishing - medaka (1 of 1)] [M::main::0.1211.18] loaded/built the index for 6 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 6 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.1261.17] distinct minimizers: 528528 (96.41% are singletons); average occurrences: 1.106; average spacing: 5.339; total length: 3122263 [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -I 16G -x map-ont -d /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta [polishing - medaka (1 of 1)] [M::main] Real time: 0.129 sec; CPU: 0.151 sec; Peak RSS: 0.032 GB [polishing - medaka (1 of 1)] [M::main::0.0201.02] loaded/built the index for 6 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_mapopt_update::0.0251.01] mid_occ = 27 [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 6 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.0291.01] distinct minimizers: 528528 (96.41% are singletons); average occurrences: 1.106; average spacing: 5.339; total length: 3122263 [polishing - medaka (1 of 1)] [M::worker_pipeline::21.74713.36] mapped 101377 sequences [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -x map-ont --secondary=no -L --MD -A 2 -B 4 -O 4,24 -E 2,1 -t 20 -a /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/READS.filt.fq.gz [polishing - medaka (1 of 1)] [M::main] Real time: 21.750 sec; CPU: 290.535 sec; Peak RSS: 1.760 GB [polishing - medaka (1 of 1)] [bam_sort_core] merging from 0 files and 20 in-memory blocks... [polishing - medaka (1 of 1)] Running medaka consensus [polishing - medaka (1 of 1)] [11:55:45 - Predict] Reducing threads to 2, anymore is a waste. [polishing - medaka (1 of 1)] [11:55:45 - Predict] It looks like you are running medaka without a GPU and attempted to set a high number of threads. We have scaled this down to an optimal number. If you wish to improve performance please see https://nanoporetech.github.io/medaka/installation.html#improving-parallelism. [polishing - medaka (1 of 1)] [11:55:45 - Predict] Setting tensorflow inter/intra-op threads to 2/1. [polishing - medaka (1 of 1)] [11:55:45 - Predict] Processing region(s): contig_1:0-2792966 contig_2:0-235945 contig_3:0-6180 contig_4:0-8803 contig_5:0-32438 contig_6:0-45931 [polishing - medaka (1 of 1)] [11:55:45 - Predict] Using model: /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/data/r941_min_sup_g507_model.tar.gz. [polishing - medaka (1 of 1)] [11:55:45 - Predict] Processing 8 long region(s) with batching. [polishing - medaka (1 of 1)] [11:55:46 - MdlStrTF] Model <keras.engine.sequential.Sequential object at 0x7fbd41bb0d30> [polishing - medaka (1 of 1)] [11:55:46 - MdlStrTF] loading weights from /tmp/tmpd821pglo/model/variables/variables [polishing - medaka (1 of 1)] [11:55:46 - BAMFile] Creating pool of 16 BAM file sets. [polishing - medaka (1 of 1)] [11:55:46 - Sampler] Initializing sampler for consensus of region contig_1:0-1000000. [polishing - medaka (1 of 1)] [11:55:46 - Sampler] Initializing sampler for consensus of region contig_1:999000-1999000. [polishing - medaka (1 of 1)] [11:55:46 - PWorker] Running inference for 3.1M draft bases. [polishing - medaka (1 of 1)] [11:55:48 - Feature] Processed contig_1:0.0-999999.4 (median depth 138.0) [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Took 1.66s to make features. [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Initializing sampler for consensus of region contig_1:1998000-2792966. [polishing - medaka (1 of 1)] [11:55:48 - Feature] Processed contig_1:999000.0-1998999.1 (median depth 120.0) [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Took 1.78s to make features. [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Initializing sampler for consensus of region contig_2:0-235945. [polishing - medaka (1 of 1)] [11:55:48 - Feature] Processed contig_2:0.0-235944.0 (median depth 168.0) [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Took 0.64s to make features. [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Initializing sampler for consensus of region contig_3:0-6180. [polishing - medaka (1 of 1)] [11:55:49 - Feature] Processed contig_3:0.0-6179.0 (median depth 327.0) [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Took 0.24s to make features. [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Initializing sampler for consensus of region contig_4:0-8803. [polishing - medaka (1 of 1)] [11:55:49 - Feature] Processed contig_1:1998000.0-2792965.0 (median depth 138.0) [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Took 0.95s to make features. [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Initializing sampler for consensus of region contig_5:0-32438. [polishing - medaka (1 of 1)] [11:55:49 - Feature] Processed contig_4:0.0-8802.0 (median depth 273.0) [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Took 0.19s to make features. [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Initializing sampler for consensus of region contig_6:0-45931. [polishing - medaka (1 of 1)] [11:55:49 - Feature] Processed contig_5:0.0-32437.0 (median depth 291.0) [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Took 0.39s to make features. [polishing - medaka (1 of 1)] [11:55:49 - Feature] Processed contig_6:0.0-45930.0 (median depth 189.0) [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Took 0.40s to make features. [polishing - medaka (1 of 1)] [11:55:58 - PWorker] Batches in cache: 7. [polishing - medaka (1 of 1)] [11:55:58 - PWorker] 14.8% Done (0.5/3.1 Mbases) in 12.1s [polishing - medaka (1 of 1)] [11:56:06 - PWorker] Batches in cache: 6. [polishing - medaka (1 of 1)] [11:56:14 - PWorker] Batches in cache: 5. [polishing - medaka (1 of 1)] [11:56:14 - PWorker] 46.2% Done (1.4/3.1 Mbases) in 28.0s [polishing - medaka (1 of 1)] [11:56:22 - PWorker] Batches in cache: 4. [polishing - medaka (1 of 1)] [11:56:30 - PWorker] Batches in cache: 3. [polishing - medaka (1 of 1)] [11:56:30 - PWorker] 76.4% Done (2.4/3.1 Mbases) in 44.0s [polishing - medaka (1 of 1)] [11:56:38 - PWorker] Batches in cache: 2. [polishing - medaka (1 of 1)] [11:56:44 - PWorker] Batches in cache: 1. [polishing - medaka (1 of 1)] [11:56:44 - PWorker] 100.0% Done (3.1/3.1 Mbases) in 57.9s [polishing - medaka (1 of 1)] [11:56:44 - PWorker] Processed 7 batches [polishing - medaka (1 of 1)] [11:56:44 - PWorker] All done, 0 remainder regions. [polishing - medaka (1 of 1)] [11:56:44 - Predict] Finished processing all regions. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] Polished assembly written to flye/polish/medaka/1/consensus.fasta, have a nice day. [dragonflye] Using assembly file 'flye/polish/medaka/1/consensus.fasta' => 'flye.fasta' [dragonflye] Assembly is 3123154, estimated genome size was 2800000 (+11.54%) [dragonflye] It contains 6 (min=500) contigs totalling 3123154 bp. [dragonflye] Getting more stats now... [dragonflye] Running: assembly-scan --transpose contigs.fa 2>&1 | sed 's/^/[stats] /' | tee -a dragonflye.log [stats] contigs.fa total_contig 6 [stats] contigs.fa total_contig_length 3123154 [stats] contigs.fa max_contig_length 2793645 [stats] contigs.fa mean_contig_length 520525 [stats] contigs.fa median_contig_length 39190 [stats] contigs.fa min_contig_length 6179 [stats] contigs.fa n50_contig_length 2793645 [stats] contigs.fa l50_contig_count 1 [stats] contigs.fa num_contig_non_acgtn 0 [stats] contigs.fa contig_percent_a 31.01 [stats] contigs.fa contig_percent_c 18.89 [stats] contigs.fa contig_percent_g 18.84 [stats] contigs.fa contig_percent_t 31.27 [stats] contigs.fa contig_percent_n 0.00 [stats] contigs.fa contig_non_acgtn 0.00 [stats] contigs.fa contigs_greater_1m 1 [stats] contigs.fa contigs_greater_100k 2 [stats] contigs.fa contigs_greater_10k 4 [stats] contigs.fa contigs_greater_1k 6 [stats] contigs.fa percent_contigs_greater_1m 16.67 [stats] contigs.fa percent_contigs_greater_100k 33.33 [stats] contigs.fa percent_contigs_greater_10k 66.67 [stats] contigs.fa percent_contigs_greater_1k 100.00 [dragonflye] Using genome graph file 'flye/assembly_graph.gfa' => 'flye-unpolished.gfa' [dragonflye] Walltime used: 9 min 40 sec [dragonflye] Results in: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1 [dragonflye] Final assembly graph: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye-unpolished.gfa [dragonflye] Final assembly contigs: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/contigs.fa [dragonflye] Dragonflye is heavily based off Shovill (https://github.com/tseemann/shovill), the Illumina counterpart [dragonflye] Done. (test-dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$

Hi @llk578496

I think this all got sorted out some time in the past. PLease feel free to reopen if not!

rpetit3 / dragonflye

Failed to run medaka consensus. - ModelStoreTF exception <class 'NotImplementedError'> #14