Closed llk578496 closed 5 months ago
Hi Eddie,
Do you mind sharing the output of conda env export
in your dragonflye environment?
Thank you! Robert
The output are as below:
name: dragonflye channels:
- conda-forge
- bioconda
- defaults
- r
- cruizperez
- liulab-dfci dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=2_gnu
- _tflow_select=2.1.0=gpu
- abseil-cpp=20211102.0=h27087fc_1
- absl-py=1.1.0=pyhd8ed1ab_0
- aiohttp=3.8.1=py38h0a891b7_1
- aiosignal=1.2.0=pyhd8ed1ab_0
- alsa-lib=1.2.6.1=h7f98852_0
- any2fasta=0.4.2=hdfd78af_3
- assembly-scan=0.4.1=pyhdfd78af_0
- astor=0.8.1=pyh9f0ad1d_0
- astunparse=1.6.3=pyhd8ed1ab_0
- async-timeout=4.0.2=pyhd8ed1ab_0
- attrs=21.4.0=pyhd8ed1ab_0
- bcftools=1.15.1=h0ea216a_0
- biopython=1.79=py38h0a891b7_2
- blinker=1.4=py_1
- brotlipy=0.7.0=py38h0a891b7_1004
- bwa=0.7.17=h7132678_9
- bzip2=1.0.8=h7f98852_4
- c-ares=1.18.1=h7f98852_0
- ca-certificates=2022.6.15=ha878542_0
- cachetools=5.0.0=pyhd8ed1ab_0
- cairo=1.16.0=ha61ee94_1011
- certifi=2022.6.15=py38h578d9bd_0
- cffi=1.15.1=py38h4a40e3a_0
- charset-normalizer=2.1.0=pyhd8ed1ab_0
- click=8.1.3=py38h578d9bd_0
- cryptography=37.0.4=py38h2b5fc30_0
- cudatoolkit=10.1.243=h8cb64d8_10
- cudnn=7.6.5.32=hc0a50b0_1
- cupti=10.1.168=0
- dragonflye=1.0.13=hdfd78af_0
- expat=2.4.8=h27087fc_0
- fastp=0.23.2=hb7a2d85_2
- flye=2.9=py38hf4f3596_1
- font-ttf-dejavu-sans-mono=2.37=hab24e00_0
- font-ttf-inconsolata=3.000=h77eed37_0
- font-ttf-source-code-pro=2.038=h77eed37_0
- font-ttf-ubuntu=0.83=hab24e00_0
- fontconfig=2.14.0=h8e229c2_0
- fonts-conda-ecosystem=1=0
- fonts-conda-forge=1=0
- freetype=2.10.4=h0708190_1
- frozenlist=1.3.0=py38h0a891b7_1
- gast=0.3.3=py_0
- gettext=0.19.8.1=h73d1719_1008
- giflib=5.2.1=h36c2ea0_2
- google-auth=2.9.1=pyh6c4a22f_0
- google-auth-oauthlib=0.4.6=pyhd8ed1ab_0
- google-pasta=0.2.0=pyh8c360ce_0
- graphite2=1.3.13=h58526e2_1001
- grpc-cpp=1.47.1=hbd84cd8_0
- grpcio=1.47.1=py38h5436ba1_0
- gsl=2.7=he838d99_0
- h5py=2.10.0=nompi_py38h9915d05_106
- harfbuzz=4.4.1=hf9f4e7c_0
- hdf5=1.10.6=nompi_h6a2412b_1114
- htslib=1.15.1=h9753748_0
- icu=70.1=h27087fc_0
- idna=3.3=pyhd8ed1ab_0
- importlib-metadata=4.11.4=py38h578d9bd_0
- intervaltree=3.0.2=py_0
- isa-l=2.30.0=ha770c72_4
- jpeg=9e=h166bdaf_2
- k8=0.2.5=hd03093a_2
- keras-preprocessing=1.1.2=pyhd8ed1ab_0
- keyutils=1.6.1=h166bdaf_0
- kmc=3.2.1=hf1761c0_2
- krb5=1.19.3=h3790be6_0
- lcms2=2.12=hddcbb42_0
- ld_impl_linux-64=2.36.1=hea4e1c9_2
- lerc=3.0=h9c3ff4c_0
- libblas=3.9.0=15_linux64_openblas
- libcblas=3.9.0=15_linux64_openblas
- libcups=2.3.3=hf5a7f15_1
- libcurl=7.83.1=h7bff187_0
- libdeflate=1.10=h7f98852_0
- libedit=3.1.20191231=he28a2e2_2
- libev=4.33=h516909a_1
- libffi=3.4.2=h7f98852_5
- libgcc-ng=12.1.0=h8d9b700_16
- libgfortran-ng=12.1.0=h69a702a_16
- libgfortran5=12.1.0=hdcd56e2_16
- libglib=2.72.1=h2d90d5f_0
- libgomp=12.1.0=h8d9b700_16
- libiconv=1.16=h516909a_0
- liblapack=3.9.0=15_linux64_openblas
- libnghttp2=1.47.0=h727a467_0
- libnsl=2.0.0=h7f98852_0
- libopenblas=0.3.20=pthreads_h78a6416_0
- libpng=1.6.37=h753d276_3
- libprotobuf=3.20.1=h6239696_0
- libssh2=1.10.0=ha56f1ee_2
- libstdcxx-ng=12.1.0=ha89aaad_16
- libtiff=4.4.0=h0fcbabc_0
- libuuid=2.32.1=h7f98852_1000
- libwebp-base=1.2.2=h7f98852_1
- libxcb=1.13=h7f98852_1004
- libzlib=1.2.12=h166bdaf_2
- lz4-c=1.9.3=h9c3ff4c_1
- mappy=2.24=py38h4c6a040_1
- markdown=3.4.1=pyhd8ed1ab_0
- medaka=1.6.1=py38h84d2cc8_0
- miniasm=0.3_r179=h7132678_3
- minimap2=2.24=h7132678_1
- multidict=6.0.2=py38h0a891b7_1
- nanoq=0.9.0=hec16e2b_0
- ncurses=6.3=h27087fc_1
- networkx=2.8.4=pyhd8ed1ab_0
- numpy=1.19.5=py38h8246c76_3
- oauthlib=3.2.0=pyhd8ed1ab_0
- ont-fast5-api=4.0.2=pyhdfd78af_0
- openjdk=11.0.15=hc6918da_0
- openssl=1.1.1q=h166bdaf_0
- opt_einsum=3.3.0=pyhd8ed1ab_1
- packaging=21.3=pyhd8ed1ab_0
- parasail-python=1.2.4=py38h3b68952_2
- pbzip2=1.1.13=0
- pcre=8.45=h9c3ff4c_0
- perl=5.32.1=2_h7f98852_perl5
- perl-file-spec=3.48_01=pl5321hdfd78af_2
- perl-findbin=1.51=pl5321hdfd78af_3
- pigz=2.6=h27826a3_0
- pilon=1.24=hdfd78af_0
- pip=22.1.2=pyhd8ed1ab_0
- pixman=0.40.0=h36c2ea0_0
- polypolish=0.5.0=h87f3376_1
- porechop=0.2.4=py38h4a32c8e_3
- progressbar33=2.4=py_0
- protobuf=3.20.1=py38hfa26641_0
- pthread-stubs=0.4=h36c2ea0_1001
- pyasn1=0.4.8=py_0
- pyasn1-modules=0.2.7=py_0
- pycparser=2.21=pyhd8ed1ab_0
- pyfaidx=0.7.0=pyh5e36f6f_0
- pyjwt=2.4.0=pyhd8ed1ab_0
- pyopenssl=22.0.0=pyhd8ed1ab_0
- pyparsing=3.0.9=pyhd8ed1ab_0
- pysam=0.19.1=py38h8bf8b8d_0
- pysocks=1.7.1=py38h578d9bd_5
- pyspoa=0.0.3=py38h8ded8fe_3
- python=3.8.13=h582c2e5_0_cpython
- python-edlib=1.3.9=py38h4a32c8e_1
- python-isal=0.11.1=py38h497a2fe_1
- python_abi=3.8=2_cp38
- pyu2f=0.1.5=pyhd8ed1ab_0
- racon=1.5.0=h7ff8a90_0
- rasusa=0.7.0=hec16e2b_0
- raven-assembler=1.8.1=h5b5514e_0
- re2=2022.06.01=h27087fc_0
- readline=8.1.2=h0f457ee_0
- requests=2.28.1=pyhd8ed1ab_0
- requests-oauthlib=1.3.1=pyhd8ed1ab_0
- rsa=4.8=pyhd8ed1ab_0
- samclip=0.4.0=hdfd78af_1
- samtools=1.15.1=h1170115_0
- scipy=1.8.1=py38h1ee437e_0
- seqtk=1.3=h7132678_4
- setuptools=63.2.0=py38h578d9bd_0
- six=1.16.0=pyh6c4a22f_0
- sortedcontainers=2.4.0=pyhd8ed1ab_0
- sqlite=3.39.1=h4ff8645_0
- tar=1.34=ha1f6473_0
- tensorboard=2.9.0=pyhd8ed1ab_0
- tensorboard-data-server=0.6.0=py38h2b5fc30_2
- tensorboard-plugin-wit=1.8.1=pyhd8ed1ab_0
- tensorflow=2.2.0=gpu_py38hb782248_0
- tensorflow-base=2.2.0=gpu_py38h83e3d50_0
- tensorflow-estimator=2.6.0=py38h709712a_0
- tensorflow-gpu=2.2.0=h0d30ee6_0
- termcolor=1.1.0=pyhd8ed1ab_3
- tk=8.6.12=h27826a3_0
- typing-extensions=4.3.0=hd8ed1ab_0
- typing_extensions=4.3.0=pyha770c72_0
- urllib3=1.26.10=pyhd8ed1ab_0
- werkzeug=2.1.2=pyhd8ed1ab_1
- whatshap=1.4=py38h4a32c8e_0
- wheel=0.37.1=pyhd8ed1ab_0
- wrapt=1.14.1=py38h0a891b7_0
- xopen=1.5.0=py38h578d9bd_0
- xorg-fixesproto=5.0=h7f98852_1002
- xorg-inputproto=2.3.2=h7f98852_1002
- xorg-kbproto=1.0.7=h7f98852_1002
- xorg-libice=1.0.10=h7f98852_0
- xorg-libsm=1.2.3=hd9c2040_1000
- xorg-libx11=1.7.2=h7f98852_0
- xorg-libxau=1.0.9=h7f98852_0
- xorg-libxdmcp=1.1.3=h7f98852_0
- xorg-libxext=1.3.4=h7f98852_1
- xorg-libxfixes=5.0.3=h7f98852_1004
- xorg-libxi=1.7.10=h7f98852_0
- xorg-libxrender=0.9.10=h7f98852_1003
- xorg-libxtst=1.2.3=h7f98852_1002
- xorg-recordproto=1.14.2=h7f98852_1002
- xorg-renderproto=0.11.1=h7f98852_1002
- xorg-xextproto=7.3.0=h7f98852_1002
- xorg-xproto=7.0.31=h7f98852_1007
- xz=5.2.5=h516909a_1
- yarl=1.7.2=py38h0a891b7_2
- zipp=3.8.0=pyhd8ed1ab_0
- zlib=1.2.12=h166bdaf_2
- zstd=1.5.2=h8a70e8d_2
Thank you!
Best regards, Eddie
Just to make sure, are expecting to run medaka on a GPU?
Only using CPU will be fine!
Try adding the CUDA_VISIBLE_DEVICES
variable to your environment, then rerunning dragonflye. This should help us determine if its due to something on the gpu side of things or not.
export CUDA_VISIBLE_DEVICES=''
We just reinstalled the dragonflye env (v1.0.13) and test run with the following commands:
conda activate dragonflye
export CUDA_VISIBLE_DEVICES=''
/mnt/data/Species-specific/CAUR$ dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --nanohq --model r941_min_sup_g507
But we seems to get the NotImplementedError
. Please see the log blow:
(base) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ conda activate dragonflye (dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ export CUDA_VISIBLE_DEVICES='' (dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA: [dragonflye] Hello gilmansiu3 [dragonflye] You ran: /home/gilmansiu3/miniconda3/envs/dragonflye/bin/dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --nanohq --model r941_min_sup_g507 [dragonflye] This is dragonflye 1.0.13 [dragonflye] Written by Robert A Petit III [dragonflye] Homepage is https://github.com/rpetit3/dragonflye [dragonflye] Operating system is linux [dragonflye] Perl version is v5.32.1 [dragonflye] Machine has 20 CPU cores and 125.72 GB RAM [dragonflye] Verifying input model (--model): r941_min_sup_g507 [dragonflye] Model r941_min_sup_g507 verified! [dragonflye] Valid model provided, but number of Medaka rounds (--medaka) not given, assuming 1 round [dragonflye] Using any2fasta - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/any2fasta | any2fasta 0.4.2 [dragonflye] Using assembly-scan - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/assembly-scan | assembly-scan 0.4.1 [dragonflye] Using bwa - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/bwa | Version: 0.7.17-r1188 [dragonflye] Using fastp - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/fastp | fastp 0.23.2 [dragonflye] Using flye - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/flye | 2.9.1-b1780 [dragonflye] Using kmc - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/kmc | K-Mer Counter (KMC) ver. 3.2.1 (2022-01-04) [dragonflye] Using medaka - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/medaka | medaka 1.6.1 [dragonflye] Using miniasm - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/miniasm | 0.3-r179 [dragonflye] Using minimap2 - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/minimap2 | 2.24-r1122 [dragonflye] Using nanoq - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/nanoq | nanoq 0.9.0 [dragonflye] Using pigz - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/pigz | pigz 2.6 [dragonflye] Using pilon - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/pilon | Pilon version 1.24 Thu Jan 28 13:00:45 2021 -0500 [dragonflye] Using polypolish - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/polypolish | Polypolish v0.5.0 [dragonflye] Using porechop - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/porechop | 0.2.4 [dragonflye] Using racon - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/racon | 1.5.0 [dragonflye] Using rasusa - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/rasusa | rasusa 0.7.0 [dragonflye] Using raven - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/raven | 1.8.1 [dragonflye] Using samclip - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/samclip | samclip 0.4.0 [dragonflye] Using samtools - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/samtools | Version: 1.15.1 (using htslib 1.15.1) [dragonflye] Using seqtk - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/seqtk | Version: 1.3-r106 [dragonflye] Using tempdir: /tmp/xaLXJ629uH [dragonflye] Changing into folder: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1 [dragonflye] Collecting raw read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 \/mnt\/data\/Species-specific\/CAUR\/02_fastq\/220818_VRE1.fastq.gz 2>&1 1>/tmp/iq_46esfn0 | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Read stats: max_len = 45857 [dragonflye] Read stats: avg_len = 4267 [dragonflye] Read stats: total_bp = 432559422 [dragonflye] Read stats: min_len = 1000 [dragonflye] Using genome size 2800000 bp [dragonflye] Estimated sequencing depth: 154x [dragonflye] Filter reads based on length and/or quality [dragonflye] Running: nanoq --min-len 1000 --input /mnt/data/Species-specific/CAUR/02_fastq/220818_VRE1.fastq.gz --min-qual 0 2>&1 1> READS.filt.fq | sed 's/^/[nanoq] /' | tee -a dragonflye.log [dragonflye] Running: pigz -f -p 20 --fast READS.filt.fq 2>&1 | sed 's/^/[pigz] /' | tee -a dragonflye.log [dragonflye] No read depth reduction requested or necessary. [dragonflye] No read adapter trimming requested. [dragonflye] Running: ln -sf READS.filt.fq.gz READS.fq.gz 2>&1 | sed 's/^/[ln] /' | tee -a dragonflye.log [dragonflye] Collecting qc'd read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 READS.fq.gz 2>&1 1>/tmp/IvNeqTcrQQ | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Final Read stats: total_bp = 432559422 [dragonflye] Final Read stats: avg_len = 4267 [dragonflye] Final Read stats: max_len = 45857 [dragonflye] Final Read stats: min_len = 1000 [dragonflye] Average read length looks like 4267 bp [dragonflye] Assembling reads with 'flye' [dragonflye] Running: flye --nano-hq READS.fq.gz -g 2800000 -i 0 --threads 20 -o flye 2>&1 | sed 's/^/[flye] /' | tee -a dragonflye.log [flye] [2022-12-01 00:38:51] INFO: Starting Flye 2.9.1-b1780 [flye] [2022-12-01 00:38:51] INFO: >>>STAGE: configure [flye] [2022-12-01 00:38:51] INFO: Configuring run [flye] [2022-12-01 00:38:56] INFO: Total read length: 432559422 [flye] [2022-12-01 00:38:56] INFO: Input genome size: 2800000 [flye] [2022-12-01 00:38:56] INFO: Estimated coverage: 154 [flye] [2022-12-01 00:38:56] INFO: Reads N50/N90: 5835 / 2000 [flye] [2022-12-01 00:38:56] INFO: Minimum overlap set to 2000 [flye] [2022-12-01 00:38:56] INFO: >>>STAGE: assembly [flye] [2022-12-01 00:38:56] INFO: Assembling disjointigs [flye] [2022-12-01 00:38:56] INFO: Reading sequences [flye] [2022-12-01 00:39:00] INFO: Building minimizer index [flye] [2022-12-01 00:39:00] INFO: Pre-calculating index storage [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 00:39:05] INFO: Filling index [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 00:39:14] INFO: Extending reads [flye] [2022-12-01 00:39:57] INFO: Overlap-based coverage: 111 [flye] [2022-12-01 00:39:57] INFO: Median overlap divergence: 0.0539394 [flye] 0% 10% 90% 100% [flye] [2022-12-01 00:40:48] INFO: Assembled 6 disjointigs [flye] [2022-12-01 00:40:48] INFO: Generating sequence [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 00:40:49] INFO: Filtering contained disjointigs [flye] 0% 10% 30% 50% 60% 80% 100% [flye] [2022-12-01 00:40:49] INFO: Contained seqs: 0 [flye] [2022-12-01 00:40:50] INFO: >>>STAGE: consensus [flye] [2022-12-01 00:40:50] INFO: Running Minimap2 [flye] [2022-12-01 00:41:16] INFO: Computing consensus [flye] [2022-12-01 00:42:56] INFO: Alignment error rate: 0.068110 [flye] [2022-12-01 00:42:56] INFO: >>>STAGE: repeat [flye] [2022-12-01 00:42:56] INFO: Building and resolving repeat graph [flye] [2022-12-01 00:42:56] INFO: Parsing disjointigs [flye] [2022-12-01 00:42:56] INFO: Building repeat graph [flye] 0% 10% 30% 50% 60% 80% 100% [flye] [2022-12-01 00:42:57] INFO: Median overlap divergence: 0.00282849 [flye] [2022-12-01 00:42:57] INFO: Parsing reads [flye] [2022-12-01 00:43:01] INFO: Aligning reads to the graph [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 00:43:10] INFO: Aligned read sequence: 383410913 / 389252928 (0.984992) [flye] [2022-12-01 00:43:10] INFO: Median overlap divergence: 0.02589 [flye] [2022-12-01 00:43:10] INFO: Mean edge coverage: 122 [flye] [2022-12-01 00:43:10] INFO: Simplifying the graph [flye] [2022-12-01 00:43:11] INFO: >>>STAGE: contigger [flye] [2022-12-01 00:43:11] INFO: Generating contigs [flye] [2022-12-01 00:43:11] INFO: Reading sequences [flye] [2022-12-01 00:43:15] INFO: Generated 9 contigs [flye] [2022-12-01 00:43:15] INFO: Added 0 scaffold connections [flye] [2022-12-01 00:43:15] INFO: >>>STAGE: finalize [flye] [2022-12-01 00:43:15] INFO: Assembly statistics: [flye] [flye] Total length: 3170131 [flye] Fragments: 9 [flye] Fragments N50: 2791741 [flye] Largest frg: 2791741 [flye] Scaffolds: 0 [flye] Mean coverage: 120 [flye] [flye] [2022-12-01 00:43:15] INFO: Final assembly: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/assembly.fasta [dragonflye] Polishing with Racon (1 rounds) [dragonflye] Running: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz 2>&1 1> flye/polish/racon/1/aligments.paf | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0571.01] collected minimizers [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0632.47] sorted minimizers [polishing - racon (1 of 1)] [M::main::0.0632.47] loaded/built the index for 9 target sequence(s) [polishing - racon (1 of 1)] [M::mm_mapopt_update::0.0682.35] mid_occ = 26 [polishing - racon (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 9 [polishing - racon (1 of 1)] [M::mm_idx_stat::0.0722.28] distinct minimizers: 534333 (95.56% are singletons); average occurrences: 1.111; average spacing: 5.339; total length: 3170131 [polishing - racon (1 of 1)] [M::worker_pipeline::11.07511.97] mapped 101377 sequences [polishing - racon (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - racon (1 of 1)] [M::main] CMD: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz [polishing - racon (1 of 1)] [M::main] Real time: 11.080 sec; CPU: 132.624 sec; Peak RSS: 0.491 GB [dragonflye] Running: racon -t 20 READS.fq.gz flye/polish/racon/1/aligments.paf flye.fasta 2>&1 1> flye/polish/racon/1/consensus.fasta | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded target sequences 0.011683 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded sequences 4.829570 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded overlaps 0.086332 s [racon::Polisher::initialize] aligning overlaps [====================] 8.289367 s ] 0.501922 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] transformed data into windows 0.464560 s [racon::Polisher::polish] generating consensus [====================] 44.015550 s ] 1.292581 s [polishing - racon (1 of 1)] [racon::Polisher::] total = 57.762712 s [dragonflye] Polishing with Medaka (1 rounds) [dragonflye] Running: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log [polishing - medaka (1 of 1)] Checking program versions [polishing - medaka (1 of 1)] This is medaka 1.6.1 [polishing - medaka (1 of 1)] Program Version Required Pass
[polishing - medaka (1 of 1)] bcftools 1.15.1 1.11 True
[polishing - medaka (1 of 1)] bgzip 1.15.1 1.11 True
[polishing - medaka (1 of 1)] minimap2 2.24 2.11 True
[polishing - medaka (1 of 1)] samtools 1.15.1 1.11 True
[polishing - medaka (1 of 1)] tabix 1.15.1 1.11 True
[polishing - medaka (1 of 1)] Aligning basecalls to draft [polishing - medaka (1 of 1)] Creating fai index file /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.fai [polishing - medaka (1 of 1)] Creating mmi index file /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.0621.03] collected minimizers [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.0731.30] sorted minimizers [polishing - medaka (1 of 1)] [M::main::0.0891.24] loaded/built the index for 8 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 8 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.0931.23] distinct minimizers: 529378 (94.98% are singletons); average occurrences: 1.122; average spacing: 5.338; total length: 3170417 [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -I 16G -x map-ont -d /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta [polishing - medaka (1 of 1)] [M::main] Real time: 0.096 sec; CPU: 0.118 sec; Peak RSS: 0.033 GB [polishing - medaka (1 of 1)] [M::main::0.0181.03] loaded/built the index for 8 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_mapopt_update::0.0231.02] mid_occ = 27 [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 8 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.0271.02] distinct minimizers: 529378 (94.98% are singletons); average occurrences: 1.122; average spacing: 5.338; total length: 3170417 [polishing - medaka (1 of 1)] [M::worker_pipeline::21.10813.95] mapped 101377 sequences [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -x map-ont --secondary=no -L --MD -A 2 -B 4 -O 4,24 -E 2,1 -t 20 -a /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/READS.filt.fq.gz [polishing - medaka (1 of 1)] [M::main] Real time: 21.111 sec; CPU: 294.369 sec; Peak RSS: 2.029 GB [polishing - medaka (1 of 1)] [bam_sort_core] merging from 0 files and 20 in-memory blocks... [polishing - medaka (1 of 1)] Running medaka consensus [polishing - medaka (1 of 1)] [00:45:01 - Predict] Reducing threads to 2, anymore is a waste. [polishing - medaka (1 of 1)] 2022-12-01 00:45:01.202626: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected [polishing - medaka (1 of 1)] [00:45:01 - Predict] It looks like you are running medaka without a GPU and attempted to set a high number of threads. We have scaled this down to an optimal number. If you wish to improve performance please see https://nanoporetech.github.io/medaka/installation.html#improving-parallelism. [polishing - medaka (1 of 1)] [00:45:01 - Predict] Setting tensorflow inter/intra-op threads to 2/1. [polishing - medaka (1 of 1)] [00:45:01 - Predict] Processing region(s): contig_1:0-2793679 contig_2:0-235943 contig_3:0-7454 contig_5:0-45936 contig_6:0-8937 contig_7:0-32438 contig_8:0-34259 contig_9:0-11771 [polishing - medaka (1 of 1)] [00:45:01 - Predict] Using model: /home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/data/r941_min_sup_g507_model.tar.gz. [polishing - medaka (1 of 1)] [00:45:01 - Predict] Processing 10 long region(s) with batching. [polishing - medaka (1 of 1)] [00:45:01 - MdlStrTF] ModelStoreTF exception <class 'NotImplementedError'> [polishing - medaka (1 of 1)] Traceback (most recent call last): [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/bin/medaka", line 11, in[polishing - medaka (1 of 1)] sys.exit(main()) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/medaka.py", line 720, in main [polishing - medaka (1 of 1)] args.func(args) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/prediction.py", line 160, in predict [polishing - medaka (1 of 1)] model = model_store.load_model(time_steps=args.chunk_len) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/datastore.py", line 159, in load_model [polishing - medaka (1 of 1)] self.model = model_partial_function(time_steps=time_steps) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/models.py", line 147, in build_model [polishing - medaka (1 of 1)] model.add(Bidirectional(gru, input_shape=input_shape)) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 456, in _method_wrapper [polishing - medaka (1 of 1)] result = method(self, args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py", line 198, in add [polishing - medaka (1 of 1)] layer(x) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/wrappers.py", line 531, in call [polishing - medaka (1 of 1)] return super(Bidirectional, self).call(inputs, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in call [polishing - medaka (1 of 1)] outputs = call_fn(cast_inputs, args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/wrappers.py", line 644, in call [polishing - medaka (1 of 1)] y = self.forward_layer(forward_inputs, [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 654, in call [polishing - medaka (1 of 1)] return super(RNN, self).call(inputs, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in call [polishing - medaka (1 of 1)] outputs = call_fn(cast_inputs, *args, *kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent_v2.py", line 408, in call [polishing - medaka (1 of 1)] inputs, initialstate, = self._process_inputs(inputs, initial_state, None) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 848, in _process_inputs [polishing - medaka (1 of 1)] initial_state = self.get_initial_state(inputs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 636, in get_initial_state [polishing - medaka (1 of 1)] init_state = get_initial_state_fn( [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 1910, in get_initial_state [polishing - medaka (1 of 1)] return _generate_zero_filled_state_for_cell(self, inputs, batch_size, dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2926, in _generate_zero_filled_state_for_cell [polishing - medaka (1 of 1)] return _generate_zero_filled_state(batch_size, cell.state_size, dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2944, in _generate_zero_filled_state [polishing - medaka (1 of 1)] return create_zeros(state_size) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2939, in create_zeros [polishing - medaka (1 of 1)] return array_ops.zeros(init_state_size, dtype=dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2677, in wrapped [polishing - medaka (1 of 1)] tensor = fun(args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2721, in zeros [polishing - medaka (1 of 1)] output = _constant_if_small(zero, shape, dtype, name) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2662, in _constant_if_small [polishing - medaka (1 of 1)] if np.prod(shape) < 1000: [polishing - medaka (1 of 1)] File "<__array_function__ internals>", line 180, in prod [polishing - medaka (1 of 1)] File "/home/gilmansiu3/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 3045, in prod [polishing - medaka (1 of 1)] return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out, [polishing - medaka (1 of 1)] File "/home/gilmansiu3/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction [polishing - medaka (1 of 1)] return ufunc.reduce(obj, axis, dtype, out, passkwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 748, in array [polishing - medaka (1 of 1)] raise NotImplementedError("Cannot convert a symbolic Tensor ({}) to a numpy" [polishing - medaka (1 of 1)] NotImplementedError: Cannot convert a symbolic Tensor (bidirectional/forward_gru1/strided_slice:0) to a numpy array. [polishing - medaka (1 of 1)] Failed to run medaka consensus. [dragonflye] Error running command: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log
Hmmmm, does flye/polish/racon/1/consensus.fasta
look like its not empty?
Yes, it is not empty.
Please see the attached file. (I renamed it with .txt
extension because Github did not support .fasta
format.)
consensus.fasta.txt
Are you able to use Dragonflye with other samples?
Yes. I tried to perform assembly on E. coli this time. But it seems to get the same error. Please find the log below.
(dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ export CUDA_VISIBLE_DEVICES='' (dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ dragonflye --reads 02_fastq/Zymo_Escherichia_coli_ATCC_25922.fastq.gz --gsize 5M --outdir 04_dragonflye/Zymo_Escherichia_coli_ATCC_25922 --cpus 20 --nanohq --model r941_min_sup_g507 [dragonflye] Hello gilmansiu3 [dragonflye] You ran: /home/gilmansiu3/miniconda3/envs/dragonflye/bin/dragonflye --reads 02_fastq/Zymo_Escherichia_coli_ATCC_25922.fastq.gz --gsize 5M --outdir 04_dragonflye/Zymo_Escherichia_coli_ATCC_25922 --cpus 20 --nanohq --model r941_min_sup_g507 [dragonflye] This is dragonflye 1.0.13 [dragonflye] Written by Robert A Petit III [dragonflye] Homepage is https://github.com/rpetit3/dragonflye [dragonflye] Operating system is linux [dragonflye] Perl version is v5.32.1 [dragonflye] Machine has 20 CPU cores and 125.72 GB RAM [dragonflye] Verifying input model (--model): r941_min_sup_g507 [dragonflye] Model r941_min_sup_g507 verified! [dragonflye] Valid model provided, but number of Medaka rounds (--medaka) not given, assuming 1 round [dragonflye] Using any2fasta - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/any2fasta | any2fasta 0.4.2 [dragonflye] Using assembly-scan - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/assembly-scan | assembly-scan 0.4.1 [dragonflye] Using bwa - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/bwa | Version: 0.7.17-r1188 [dragonflye] Using fastp - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/fastp | fastp 0.23.2 [dragonflye] Using flye - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/flye | 2.9.1-b1780 [dragonflye] Using kmc - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/kmc | K-Mer Counter (KMC) ver. 3.2.1 (2022-01-04) [dragonflye] Using medaka - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/medaka | medaka 1.6.1 [dragonflye] Using miniasm - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/miniasm | 0.3-r179 [dragonflye] Using minimap2 - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/minimap2 | 2.24-r1122 [dragonflye] Using nanoq - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/nanoq | nanoq 0.9.0 [dragonflye] Using pigz - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/pigz | pigz 2.6 [dragonflye] Using pilon - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/pilon | Pilon version 1.24 Thu Jan 28 13:00:45 2021 -0500 [dragonflye] Using polypolish - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/polypolish | Polypolish v0.5.0 [dragonflye] Using porechop - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/porechop | 0.2.4 [dragonflye] Using racon - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/racon | 1.5.0 [dragonflye] Using rasusa - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/rasusa | rasusa 0.7.0 [dragonflye] Using raven - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/raven | 1.8.1 [dragonflye] Using samclip - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/samclip | samclip 0.4.0 [dragonflye] Using samtools - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/samtools | Version: 1.15.1 (using htslib 1.15.1) [dragonflye] Using seqtk - /home/gilmansiu3/miniconda3/envs/dragonflye/bin/seqtk | Version: 1.3-r106 [dragonflye] Using tempdir: /tmp/s8fFyOxwj0 [dragonflye] Changing into folder: /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922 [dragonflye] Collecting raw read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 \/mnt\/data\/Species-specific\/CAUR\/02_fastq\/Zymo_Escherichia_coli_ATCC_25922.fastq.gz 2>&1 1>/tmp/cJKCrPwvUV | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Read stats: max_len = 79759 [dragonflye] Read stats: avg_len = 6288 [dragonflye] Read stats: total_bp = 1196562344 [dragonflye] Read stats: min_len = 1006 [dragonflye] Using genome size 5000000 bp [dragonflye] Estimated sequencing depth: 239x [dragonflye] Filter reads based on length and/or quality [dragonflye] Running: nanoq --min-len 1000 --input /mnt/data/Species-specific/CAUR/02_fastq/Zymo_Escherichia_coli_ATCC_25922.fastq.gz --min-qual 0 2>&1 1> READS.filt.fq | sed 's/^/[nanoq] /' | tee -a dragonflye.log [dragonflye] Running: pigz -f -p 20 --fast READS.filt.fq 2>&1 | sed 's/^/[pigz] /' | tee -a dragonflye.log [dragonflye] Subsampling reads from 239x to 150x [dragonflye] Running: rasusa -i READS.filt.fq.gz -c 150 -g 5000000 -s 42 2>&1 1> READS.sub.fq | sed 's/^/[rasusa] /' | tee -a dragonflye.log [rasusa] [2022-12-01][16:53:37][rasusa][INFO] Target number of bases to subsample to is: 750000000 [rasusa] [2022-12-01][16:53:37][rasusa][INFO] Gathering read lengths... [rasusa] [2022-12-01][16:53:46][rasusa][INFO] 190296 reads detected [rasusa] [2022-12-01][16:53:46][rasusa][INFO] Keeping 118951 reads [rasusa] [2022-12-01][16:53:56][rasusa][INFO] Actual coverage of kept reads is 150.00x [rasusa] [2022-12-01][16:53:56][rasusa][INFO] Done π [dragonflye] Running: pigz -f -p 20 --fast READS.sub.fq 2>&1 | sed 's/^/[pigz] /' | tee -a dragonflye.log [dragonflye] No read adapter trimming requested. [dragonflye] Running: ln -sf READS.sub.fq.gz READS.fq.gz 2>&1 | sed 's/^/[ln] /' | tee -a dragonflye.log [dragonflye] Collecting qc'd read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 READS.fq.gz 2>&1 1>/tmp/qdiQ_mghtw | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Final Read stats: avg_len = 6305 [dragonflye] Final Read stats: min_len = 1006 [dragonflye] Final Read stats: total_bp = 750001566 [dragonflye] Final Read stats: max_len = 79759 [dragonflye] Average read length looks like 6305 bp [dragonflye] Assembling reads with 'flye' [dragonflye] Running: flye --nano-hq READS.fq.gz -g 5000000 -i 0 --threads 20 -o flye 2>&1 | sed 's/^/[flye] /' | tee -a dragonflye.log [flye] [2022-12-01 16:54:08] INFO: Starting Flye 2.9.1-b1780 [flye] [2022-12-01 16:54:08] INFO: >>>STAGE: configure [flye] [2022-12-01 16:54:08] INFO: Configuring run [flye] [2022-12-01 16:54:17] INFO: Total read length: 750001566 [flye] [2022-12-01 16:54:17] INFO: Input genome size: 5000000 [flye] [2022-12-01 16:54:17] INFO: Estimated coverage: 150 [flye] [2022-12-01 16:54:17] INFO: Reads N50/N90: 9215 / 2848 [flye] [2022-12-01 16:54:17] INFO: Minimum overlap set to 3000 [flye] [2022-12-01 16:54:17] INFO: >>>STAGE: assembly [flye] [2022-12-01 16:54:17] INFO: Assembling disjointigs [flye] [2022-12-01 16:54:17] INFO: Reading sequences [flye] [2022-12-01 16:54:25] INFO: Building minimizer index [flye] [2022-12-01 16:54:25] INFO: Pre-calculating index storage [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 16:54:32] INFO: Filling index [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 16:54:51] INFO: Extending reads [flye] [2022-12-01 16:56:14] INFO: Overlap-based coverage: 120 [flye] [2022-12-01 16:56:14] INFO: Median overlap divergence: 0.0726324 [flye] 0% 90% 100% [flye] [2022-12-01 16:58:21] INFO: Assembled 5 disjointigs [flye] [2022-12-01 16:58:22] INFO: Generating sequence [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 16:58:24] INFO: Filtering contained disjointigs [flye] 0% 20% 40% 60% 80% 100% [flye] [2022-12-01 16:58:25] INFO: Contained seqs: 0 [flye] [2022-12-01 16:58:25] INFO: >>>STAGE: consensus [flye] [2022-12-01 16:58:25] INFO: Running Minimap2 [flye] [2022-12-01 16:59:07] INFO: Computing consensus [flye] [2022-12-01 17:00:16] INFO: Alignment error rate: 0.081555 [flye] [2022-12-01 17:00:16] INFO: >>>STAGE: repeat [flye] [2022-12-01 17:00:16] INFO: Building and resolving repeat graph [flye] [2022-12-01 17:00:16] INFO: Parsing disjointigs [flye] [2022-12-01 17:00:16] INFO: Building repeat graph [flye] 0% 20% 40% 60% 80% 100% [flye] [2022-12-01 17:00:18] INFO: Median overlap divergence: 0.0124352 [flye] [2022-12-01 17:00:18] INFO: Parsing reads [flye] [2022-12-01 17:00:25] INFO: Aligning reads to the graph [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-01 17:00:41] INFO: Aligned read sequence: 654011637 / 667036201 (0.980474) [flye] [2022-12-01 17:00:41] INFO: Median overlap divergence: 0.0368736 [flye] [2022-12-01 17:00:41] INFO: Mean edge coverage: 126 [flye] [2022-12-01 17:00:41] INFO: Simplifying the graph [flye] [2022-12-01 17:00:42] INFO: >>>STAGE: contigger [flye] [2022-12-01 17:00:42] INFO: Generating contigs [flye] [2022-12-01 17:00:42] INFO: Reading sequences [flye] [2022-12-01 17:00:49] INFO: Generated 5 contigs [flye] [2022-12-01 17:00:49] INFO: Added 0 scaffold connections [flye] [2022-12-01 17:00:50] INFO: >>>STAGE: finalize [flye] [2022-12-01 17:00:50] INFO: Assembly statistics: [flye] [flye] Total length: 5208142 [flye] Fragments: 5 [flye] Fragments N50: 5129207 [flye] Largest frg: 5129207 [flye] Scaffolds: 0 [flye] Mean coverage: 129 [flye] [flye] [2022-12-01 17:00:50] INFO: Final assembly: /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/assembly.fasta [dragonflye] Polishing with Racon (1 rounds) [dragonflye] Running: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz 2>&1 1> flye/polish/racon/1/aligments.paf | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0891.01] collected minimizers [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0992.68] sorted minimizers [polishing - racon (1 of 1)] [M::main::0.0992.68] loaded/built the index for 5 target sequence(s) [polishing - racon (1 of 1)] [M::mm_mapopt_update::0.1092.53] mid_occ = 10 [polishing - racon (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 5 [polishing - racon (1 of 1)] [M::mm_idx_stat::0.1172.43] distinct minimizers: 942970 (97.58% are singletons); average occurrences: 1.032; average spacing: 5.351; total length: 5208142 [polishing - racon (1 of 1)] [M::worker_pipeline::9.0619.33] mapped 78175 sequences [polishing - racon (1 of 1)] [M::worker_pipeline::11.40711.26] mapped 40776 sequences [polishing - racon (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - racon (1 of 1)] [M::main] CMD: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz [polishing - racon (1 of 1)] [M::main] Real time: 11.412 sec; CPU: 128.446 sec; Peak RSS: 0.807 GB [dragonflye] Running: racon -t 20 READS.fq.gz flye/polish/racon/1/aligments.paf flye.fasta 2>&1 1> flye/polish/racon/1/consensus.fasta | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded target sequences 0.021195 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded sequences 8.468119 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded overlaps 0.106399 s [racon::Polisher::initialize] aligning overlaps [====================] 19.257848 s ] 1.160865 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] transformed data into windows 0.800375 s [racon::Polisher::polish] generating consensus [====================] 83.666797 s ] 4.421268 s [polishing - racon (1 of 1)] [racon::Polisher::] total = 112.396382 s [dragonflye] Polishing with Medaka (1 rounds) [dragonflye] Running: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log [polishing - medaka (1 of 1)] Checking program versions [polishing - medaka (1 of 1)] This is medaka 1.6.1 [polishing - medaka (1 of 1)] Program Version Required Pass
[polishing - medaka (1 of 1)] bcftools 1.15.1 1.11 True
[polishing - medaka (1 of 1)] bgzip 1.15.1 1.11 True
[polishing - medaka (1 of 1)] minimap2 2.24 2.11 True
[polishing - medaka (1 of 1)] samtools 1.15.1 1.11 True
[polishing - medaka (1 of 1)] tabix 1.15.1 1.11 True
[polishing - medaka (1 of 1)] Aligning basecalls to draft [polishing - medaka (1 of 1)] Creating fai index file /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/polish/racon/1/consensus.fasta.fai [polishing - medaka (1 of 1)] Creating mmi index file /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/polish/racon/1/consensus.fasta.map-ont.mmi [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.1411.02] collected minimizers [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.1621.27] sorted minimizers [polishing - medaka (1 of 1)] [M::main::0.1901.23] loaded/built the index for 5 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 5 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.1981.22] distinct minimizers: 939300 (97.39% are singletons); average occurrences: 1.037; average spacing: 5.350; total length: 5209708 [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -I 16G -x map-ont -d /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/polish/racon/1/consensus.fasta [polishing - medaka (1 of 1)] [M::main] Real time: 0.201 sec; CPU: 0.244 sec; Peak RSS: 0.053 GB [polishing - medaka (1 of 1)] [M::main::0.0351.01] loaded/built the index for 5 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_mapopt_update::0.0441.01] mid_occ = 10 [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 5 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.0521.01] distinct minimizers: 939300 (97.39% are singletons); average occurrences: 1.037; average spacing: 5.350; total length: 5209708 [polishing - medaka (1 of 1)] [M::worker_pipeline::26.37714.81] mapped 78175 sequences [polishing - medaka (1 of 1)] [M::worker_pipeline::27.70514.15] mapped 40776 sequences [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -x map-ont --secondary=no -L --MD -A 2 -B 4 -O 4,24 -E 2,1 -t 20 -a /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/Zymo_Escherichia_coli_ATCC_25922/READS.sub.fq.gz [polishing - medaka (1 of 1)] [M::main] Real time: 27.743 sec; CPU: 392.046 sec; Peak RSS: 2.452 GB [polishing - medaka (1 of 1)] [bam_sort_core] merging from 0 files and 20 in-memory blocks... [polishing - medaka (1 of 1)] Running medaka consensus [polishing - medaka (1 of 1)] [17:03:44 - Predict] Reducing threads to 2, anymore is a waste. [polishing - medaka (1 of 1)] 2022-12-01 17:03:44.334565: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected [polishing - medaka (1 of 1)] [17:03:44 - Predict] It looks like you are running medaka without a GPU and attempted to set a high number of threads. We have scaled this down to an optimal number. If you wish to improve performance please see https://nanoporetech.github.io/medaka/installation.html#improving-parallelism. [polishing - medaka (1 of 1)] [17:03:44 - Predict] Setting tensorflow inter/intra-op threads to 2/1. [polishing - medaka (1 of 1)] [17:03:44 - Predict] Processing region(s): contig_1:0-5131525 contig_2:0-2989 contig_3:0-5557 contig_4:0-45470 contig_5:0-24167 [polishing - medaka (1 of 1)] [17:03:44 - Predict] Using model: /home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/data/r941_min_sup_g507_model.tar.gz. [polishing - medaka (1 of 1)] [17:03:44 - Predict] Processing 10 long region(s) with batching. [polishing - medaka (1 of 1)] [17:03:44 - MdlStrTF] ModelStoreTF exception <class 'NotImplementedError'> [polishing - medaka (1 of 1)] Traceback (most recent call last): [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/bin/medaka", line 11, in[polishing - medaka (1 of 1)] sys.exit(main()) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/medaka.py", line 720, in main [polishing - medaka (1 of 1)] args.func(args) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/prediction.py", line 160, in predict [polishing - medaka (1 of 1)] model = model_store.load_model(time_steps=args.chunk_len) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/datastore.py", line 159, in load_model [polishing - medaka (1 of 1)] self.model = model_partial_function(time_steps=time_steps) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/medaka/models.py", line 147, in build_model [polishing - medaka (1 of 1)] model.add(Bidirectional(gru, input_shape=input_shape)) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 456, in _method_wrapper [polishing - medaka (1 of 1)] result = method(self, args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py", line 198, in add [polishing - medaka (1 of 1)] layer(x) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/wrappers.py", line 531, in call [polishing - medaka (1 of 1)] return super(Bidirectional, self).call(inputs, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in call [polishing - medaka (1 of 1)] outputs = call_fn(cast_inputs, args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/wrappers.py", line 644, in call [polishing - medaka (1 of 1)] y = self.forward_layer(forward_inputs, [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 654, in call [polishing - medaka (1 of 1)] return super(RNN, self).call(inputs, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in call [polishing - medaka (1 of 1)] outputs = call_fn(cast_inputs, *args, *kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent_v2.py", line 408, in call [polishing - medaka (1 of 1)] inputs, initialstate, = self._process_inputs(inputs, initial_state, None) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 848, in _process_inputs [polishing - medaka (1 of 1)] initial_state = self.get_initial_state(inputs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 636, in get_initial_state [polishing - medaka (1 of 1)] init_state = get_initial_state_fn( [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 1910, in get_initial_state [polishing - medaka (1 of 1)] return _generate_zero_filled_state_for_cell(self, inputs, batch_size, dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2926, in _generate_zero_filled_state_for_cell [polishing - medaka (1 of 1)] return _generate_zero_filled_state(batch_size, cell.state_size, dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2944, in _generate_zero_filled_state [polishing - medaka (1 of 1)] return create_zeros(state_size) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 2939, in create_zeros [polishing - medaka (1 of 1)] return array_ops.zeros(init_state_size, dtype=dtype) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2677, in wrapped [polishing - medaka (1 of 1)] tensor = fun(args, kwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2721, in zeros [polishing - medaka (1 of 1)] output = _constant_if_small(zero, shape, dtype, name) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py", line 2662, in _constant_if_small [polishing - medaka (1 of 1)] if np.prod(shape) < 1000: [polishing - medaka (1 of 1)] File "<__array_function__ internals>", line 180, in prod [polishing - medaka (1 of 1)] File "/home/gilmansiu3/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 3045, in prod [polishing - medaka (1 of 1)] return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out, [polishing - medaka (1 of 1)] File "/home/gilmansiu3/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction [polishing - medaka (1 of 1)] return ufunc.reduce(obj, axis, dtype, out, passkwargs) [polishing - medaka (1 of 1)] File "/home/gilmansiu3/miniconda3/envs/dragonflye/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 748, in array [polishing - medaka (1 of 1)] raise NotImplementedError("Cannot convert a symbolic Tensor ({}) to a numpy" [polishing - medaka (1 of 1)] NotImplementedError: Cannot convert a symbolic Tensor (bidirectional/forward_gru1/strided_slice:0) to a numpy array. [polishing - medaka (1 of 1)] Failed to run medaka consensus. [dragonflye] Error running command: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log (dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$
I wonder what would happen if you used a different version of medaka.
Maybe try:
mamba create -n test-dragonflye -c conda-forge -c bioconda dragonflye 'medaka>1.6.1'
I just tried to create the test-dragonflye
env with medaka>1.6.1
, but seems it could only get dragonflye=1.0.7
.
(base) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ mamba create -n test-dragonflye -c conda-forge -c bioconda dragonflye 'medaka>1.6.1'
__ __ __ __ / \ / \ / \ / \ / \/ \/ \/ \
βββββββββββββββ/ /ββ/ /ββ/ /ββ/ /ββββββββββββββββββββββββ / / \ / \ / \ / \ _ / / _/ _/ _/ \ o _, / _/ ___/ ` |/ ββββ ββββ ββββββ ββββ βββββββββββ ββββββ βββββ ββββββββββββββββββ βββββββββββββββββββββ ββββββββββββββββββββββββββββββββββββββββββββββ ββββββββββββββββββββββββββββββββββββββββββββββ βββ βββ ββββββ ββββββ βββ ββββββββββββββ βββ βββ ββββββ ββββββ ββββββββββ βββ βββ
mamba (0.22.1) supported by @QuantStack GitHub: https://github.com/mamba-org/mamba Twitter: https://twitter.com/QuantStack
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Looking for: ['dragonflye', "medaka[version='>1.6.1']"]
conda-forge/linux-64 Using cache conda-forge/noarch Using cache bioconda/linux-64 Using cache bioconda/noarch Using cache r/linux-64 Using cache r/noarch Using cache pkgs/r/linux-64 No change pkgs/r/noarch No change pkgs/main/linux-64 No change pkgs/main/noarch No change cruizperez/linux-64 No change liulab-dfci/noarch No change cruizperez/noarch No change liulab-dfci/linux-64 No change Transaction
Prefix: /home/gilmansiu3/miniconda3/envs/test-dragonflye
Updating specs:
- dragonflye
medaka[version='>1.6.1']
Package Version Build Channel Size ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ Install: ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
- _libgcc_mutex 0.1 conda_forge conda-forge/linux-64 Cached
- _openmp_mutex 4.5 2_gnu conda-forge/linux-64 Cached
- abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge/linux-64 Cached
- absl-py 1.3.0 pyhd8ed1ab_0 conda-forge/noarch Cached
- aiohttp 3.8.3 py38h0a891b7_1 conda-forge/linux-64 Cached
- aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge/noarch Cached
- any2fasta 0.4.2 hdfd78af_3 bioconda/noarch Cached
- assembly-scan 0.4.1 pyhdfd78af_0 bioconda/noarch Cached
- astunparse 1.6.3 pyhd8ed1ab_0 conda-forge/noarch Cached
- async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge/noarch Cached
- attrs 22.1.0 pyh71513ae_1 conda-forge/noarch Cached
- bcftools 1.16 hfe4b78e_1 bioconda/linux-64 Cached
- biopython 1.80 py38h0a891b7_0 conda-forge/linux-64 Cached
- blinker 1.5 pyhd8ed1ab_0 conda-forge/noarch Cached
- brotlipy 0.7.0 py38h0a891b7_1005 conda-forge/linux-64 Cached
- bzip2 1.0.8 h7f98852_4 conda-forge/linux-64 Cached
- c-ares 1.18.1 h7f98852_0 conda-forge/linux-64 Cached
- ca-certificates 2022.9.24 ha878542_0 conda-forge/linux-64 Cached
- cached-property 1.5.2 hd8ed1ab_1 conda-forge/noarch Cached
- cached_property 1.5.2 pyha770c72_1 conda-forge/noarch Cached
- cachetools 4.2.4 pyhd8ed1ab_0 conda-forge/noarch Cached
- certifi 2022.9.24 pyhd8ed1ab_0 conda-forge/noarch Cached
- cffi 1.15.1 py38h4a40e3a_2 conda-forge/linux-64 Cached
- charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge/noarch Cached
- click 8.1.3 unix_pyhd8ed1ab_2 conda-forge/noarch Cached
- cryptography 38.0.4 py38h2b5fc30_0 conda-forge/linux-64 Cached
- dragonflye 1.0.7 hdfd78af_0 bioconda/noarch Cached
- flye 2.9.1 py38hf4f3596_0 bioconda/linux-64 Cached
- frozenlist 1.3.3 py38h0a891b7_0 conda-forge/linux-64 Cached
- gast 0.4.0 pyh9f0ad1d_0 conda-forge/noarch Cached
- giflib 5.2.1 h36c2ea0_2 conda-forge/linux-64 Cached
- google-auth 1.35.0 pyh6c4a22f_0 conda-forge/noarch Cached
- google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge/noarch Cached
- google-pasta 0.2.0 pyh8c360ce_0 conda-forge/noarch Cached
- grpc-cpp 1.43.2 h9e046d8_3 conda-forge/linux-64 Cached
- grpcio 1.43.0 py38hdd6454d_0 conda-forge/linux-64 Cached
- gsl 2.7 he838d99_0 conda-forge/linux-64 Cached
- h5py 3.7.0 nompi_py38h045baee_101 conda-forge/linux-64 Cached
- hdf5 1.12.2 nompi_h2386368_100 conda-forge/linux-64 Cached
- htslib 1.16 h6bc39ce_0 bioconda/linux-64 Cached
- icu 70.1 h27087fc_0 conda-forge/linux-64 Cached
- idna 3.4 pyhd8ed1ab_0 conda-forge/noarch Cached
- importlib-metadata 5.1.0 pyha770c72_0 conda-forge/noarch Cached
- intervaltree 3.0.2 py_0 conda-forge/noarch Cached
- jpeg 9e h166bdaf_2 conda-forge/linux-64 Cached
- k8 0.2.5 hd03093a_2 bioconda/linux-64 Cached
- keras 2.7.0 pyhd8ed1ab_0 conda-forge/noarch Cached
- keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge/noarch Cached
- keyutils 1.6.1 h166bdaf_0 conda-forge/linux-64 Cached
- kmc 3.2.1 hf1761c0_2 bioconda/linux-64 Cached
- krb5 1.19.3 h3790be6_0 conda-forge/linux-64 Cached
- ld_impl_linux-64 2.39 hcc3a1bd_1 conda-forge/linux-64 Cached
- libblas 3.9.0 16_linux64_openblas conda-forge/linux-64 Cached
- libcblas 3.9.0 16_linux64_openblas conda-forge/linux-64 Cached
- libcurl 7.86.0 h7bff187_1 conda-forge/linux-64 Cached
- libdeflate 1.13 h166bdaf_0 conda-forge/linux-64 Cached
- libedit 3.1.20191231 he28a2e2_2 conda-forge/linux-64 Cached
- libev 4.33 h516909a_1 conda-forge/linux-64 Cached
- libffi 3.4.2 h7f98852_5 conda-forge/linux-64 Cached
- libgcc-ng 12.2.0 h65d4601_19 conda-forge/linux-64 Cached
- libgfortran-ng 12.2.0 h69a702a_19 conda-forge/linux-64 Cached
- libgfortran5 12.2.0 h337968e_19 conda-forge/linux-64 Cached
- libgomp 12.2.0 h65d4601_19 conda-forge/linux-64 Cached
- libiconv 1.17 h166bdaf_0 conda-forge/linux-64 Cached
- liblapack 3.9.0 16_linux64_openblas conda-forge/linux-64 Cached
- libnghttp2 1.47.0 hdcd2b5c_1 conda-forge/linux-64 Cached
- libnsl 2.0.0 h7f98852_0 conda-forge/linux-64 Cached
- libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge/linux-64 Cached
- libpng 1.6.39 h753d276_0 conda-forge/linux-64 Cached
- libprotobuf 3.19.4 h780b84a_0 conda-forge/linux-64 Cached
- libsqlite 3.40.0 h753d276_0 conda-forge/linux-64 Cached
- libssh2 1.10.0 haa6b8db_3 conda-forge/linux-64 Cached
- libstdcxx-ng 12.2.0 h46fd767_19 conda-forge/linux-64 Cached
- libuuid 2.32.1 h7f98852_1000 conda-forge/linux-64 Cached
- libzlib 1.2.13 h166bdaf_4 conda-forge/linux-64 Cached
- mappy 2.24 py38h4c6a040_1 bioconda/linux-64 Cached
- markdown 3.4.1 pyhd8ed1ab_0 conda-forge/noarch Cached
- markupsafe 2.1.1 py38h0a891b7_2 conda-forge/linux-64 Cached
- medaka 1.7.2 py38h6239ad4_1 bioconda/linux-64 Cached
- miniasm 0.3_r179 h7132678_3 bioconda/linux-64 Cached
- minimap2 2.24 h7132678_1 bioconda/linux-64 Cached
- multidict 6.0.2 py38h0a891b7_2 conda-forge/linux-64 Cached
- nanoq 0.9.0 hec16e2b_1 bioconda/linux-64 Cached
- ncurses 6.3 h27087fc_1 conda-forge/linux-64 Cached
- numpy 1.19.5 py38h8246c76_3 conda-forge/linux-64 Cached
- oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge/noarch Cached
- ont-fast5-api 4.1.0 pyhdfd78af_0 bioconda/noarch Cached
- openssl 1.1.1s h0b41bf4_1 conda-forge/linux-64 Cached
- opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge/noarch Cached
- packaging 21.3 pyhd8ed1ab_0 conda-forge/noarch Cached
- parasail-python 1.3.3 py38h3b68952_0 bioconda/linux-64 Cached
- perl 5.32.1 2_h7f98852_perl5 conda-forge/linux-64 Cached
- perl-file-spec 3.48_01 pl5321hdfd78af_2 bioconda/noarch Cached
- perl-findbin 1.51 pl5321hdfd78af_3 bioconda/noarch Cached
- pigz 2.6 h27826a3_0 conda-forge/linux-64 Cached
- pip 22.3.1 pyhd8ed1ab_0 conda-forge/noarch Cached
- progressbar33 2.4 py_0 conda-forge/noarch Cached
- protobuf 3.19.4 py38h709712a_0 conda-forge/linux-64 Cached
- pyasn1 0.4.8 py_0 conda-forge/noarch Cached
- pyasn1-modules 0.2.7 py_0 conda-forge/noarch Cached
- pycparser 2.21 pyhd8ed1ab_0 conda-forge/noarch Cached
- pyjwt 2.6.0 pyhd8ed1ab_0 conda-forge/noarch Cached
- pyopenssl 22.1.0 pyhd8ed1ab_0 conda-forge/noarch Cached
- pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge/noarch Cached
- pysam 0.19.1 py38h1c8baaf_1 bioconda/linux-64 Cached
- pysocks 1.7.1 pyha2e5f31_6 conda-forge/noarch Cached
- pyspoa 0.0.3 py38h8ded8fe_3 bioconda/linux-64 Cached
- python 3.8.15 h257c98d_0_cpython conda-forge/linux-64 Cached
- python-edlib 1.3.9 py38h4a32c8e_1 bioconda/linux-64 Cached
- python-flatbuffers 2.0 pyhd8ed1ab_0 conda-forge/noarch Cached
- python_abi 3.8 3_cp38 conda-forge/linux-64 Cached
- pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge/noarch Cached
- racon 1.5.0 h7ff8a90_0 bioconda/linux-64 Cached
- rasusa 0.7.0 hec16e2b_1 bioconda/linux-64 Cached
- raven-assembler 1.8.1 h5b5514e_1 bioconda/linux-64 Cached
- re2 2022.02.01 h9c3ff4c_0 conda-forge/linux-64 Cached
- readline 8.1.2 h0f457ee_0 conda-forge/linux-64 Cached
- requests 2.28.1 pyhd8ed1ab_1 conda-forge/noarch Cached
- requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge/noarch Cached
- rsa 4.9 pyhd8ed1ab_0 conda-forge/noarch Cached
- samtools 1.16.1 h6899075_1 bioconda/linux-64 Cached
- scipy 1.9.1 py38hea3f02b_0 conda-forge/linux-64 Cached
- seqtk 1.3 h7132678_4 bioconda/linux-64 Cached
- setuptools 65.5.1 pyhd8ed1ab_0 conda-forge/noarch Cached
- six 1.16.0 pyh6c4a22f_0 conda-forge/noarch Cached
- snappy 1.1.9 hbd366e4_2 conda-forge/linux-64 Cached
- sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge/noarch Cached
- sqlite 3.40.0 h4ff8645_0 conda-forge/linux-64 Cached
- tar 1.34 hb2e2bae_1 conda-forge/linux-64 Cached
- tensorboard 2.6.0 pyhd8ed1ab_1 conda-forge/noarch Cached
- tensorboard-data-server 0.6.1 py38h2b5fc30_4 conda-forge/linux-64 Cached
- tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge/noarch Cached
- tensorflow 2.7.1 cpu_py38h66f0ec1_0 conda-forge/linux-64 Cached
- tensorflow-base 2.7.1 cpu_py38ha28dbe6_0 conda-forge/linux-64 Cached
- tensorflow-estimator 2.7.1 cpu_py38h4e23bc6_0 conda-forge/linux-64 Cached
- termcolor 2.1.1 pyhd8ed1ab_0 conda-forge/noarch Cached
- tk 8.6.12 h27826a3_0 conda-forge/linux-64 Cached
- typing-extensions 4.4.0 hd8ed1ab_0 conda-forge/noarch Cached
- typing_extensions 4.4.0 pyha770c72_0 conda-forge/noarch Cached
- urllib3 1.26.13 pyhd8ed1ab_0 conda-forge/noarch Cached
- werkzeug 2.2.2 pyhd8ed1ab_0 conda-forge/noarch Cached
- wheel 0.38.4 pyhd8ed1ab_0 conda-forge/noarch Cached
- wrapt 1.14.1 py38h0a891b7_1 conda-forge/linux-64 Cached
- xz 5.2.6 h166bdaf_0 conda-forge/linux-64 Cached
- yarl 1.8.1 py38h0a891b7_0 conda-forge/linux-64 Cached
- zipp 3.11.0 pyhd8ed1ab_0 conda-forge/noarch Cached
- zlib 1.2.13 h166bdaf_4 conda-forge/linux-64 Cached
- zstd 1.5.2 h6239696_4 conda-forge/linux-64 Cached
Summary:
Install: 148 packages
Total download: 0 B
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Confirm changes: [Y/n] y Preparing transaction: done Verifying transaction: done Executing transaction: done
To activate this environment, use
$ mamba activate test-dragonflye
To deactivate an active environment, use
$ mamba deactivate
(base) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$
Also, I just tried to use the test-dragonflye
env with dragonflye=1.0.7 & medaka=1.7.2
and it works. But it could not support the option --nanohq
as it is using an older version of dragonflye
.
(base) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ conda activate test-dragonflye (test-dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$ dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --model r941_min_sup_g507 [dragonflye] Hello gilmansiu3 [dragonflye] You ran: /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --model r941_min_sup_g507 [dragonflye] This is dragonflye 1.0.7 [dragonflye] Written by Robert A Petit III [dragonflye] Homepage is https://github.com/rpetit3/dragonflye [dragonflye] Operating system is linux [dragonflye] Perl version is v5.32.1 [dragonflye] Machine has 20 CPU cores and 125.72 GB RAM [dragonflye] Verifying input model (--model): r941_min_sup_g507 [dragonflye] Model r941_min_sup_g507 verified! [dragonflye] Valid model provided, but number of Medaka rounds (--medaka) not given, assuming 1 round [dragonflye] Using any2fasta - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/any2fasta | any2fasta 0.4.2 [dragonflye] Using assembly-scan - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/assembly-scan | assembly-scan 0.4.1 [dragonflye] Using flye - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/flye | 2.9.1-b1780 [dragonflye] Using kmc - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/kmc | K-Mer Counter (KMC) ver. 3.2.1 (2022-01-04) [dragonflye] Using medaka - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/medaka | medaka 1.7.2 [dragonflye] Using miniasm - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/miniasm | 0.3-r179 [dragonflye] Using minimap2 - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/minimap2 | 2.24-r1122 [dragonflye] Using nanoq - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/nanoq | nanoq 0.9.0 [dragonflye] Using pigz - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/pigz | pigz 2.6 [dragonflye] Using racon - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/racon | 1.5.0 [dragonflye] Using rasusa - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/rasusa | rasusa 0.7.0 [dragonflye] Using raven - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/raven | 1.8.1 [dragonflye] Using seqtk - /home/gilmansiu3/miniconda3/envs/test-dragonflye/bin/seqtk | Version: 1.3-r106 [dragonflye] Using tempdir: /tmp/vmMKaDqOh4 [dragonflye] Changing into folder: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1 [dragonflye] Collecting raw read statistics with 'seqtk' [dragonflye] Running: seqtk fqchk -q3 \/mnt\/data\/Species-specific\/CAUR\/02_fastq\/220818_VRE1.fastq.gz 2>&1 1>/tmp/RGfzcT9ipW | sed 's/^/[seqtk] /' | tee -a dragonflye.log [dragonflye] Read stats: max_len = 45857 [dragonflye] Read stats: total_bp = 432559422 [dragonflye] Read stats: avg_len = 4267 [dragonflye] Read stats: min_len = 1000 [dragonflye] Using genome size 2800000 bp [dragonflye] Estimated sequencing depth: 154x [dragonflye] Filter reads based on length and/or quality [dragonflye] Running: nanoq --min-len 1000 --input /mnt/data/Species-specific/CAUR/02_fastq/220818_VRE1.fastq.gz --min-qual 0 2>&1 1> READS.filt.fq | sed 's/^/[nanoq] /' | tee -a dragonflye.log [dragonflye] Running: pigz -f -p 20 --fast READS.filt.fq 2>&1 | sed 's/^/[pigz] /' | tee -a dragonflye.log [dragonflye] No read depth reduction requested or necessary. [dragonflye] Running: ln -sf READS.filt.fq.gz READS.fq.gz 2>&1 | sed 's/^/[ln] /' | tee -a dragonflye.log [dragonflye] Average read length looks like 4267 bp [dragonflye] Assembling reads with 'flye' [dragonflye] Running: flye --nano-raw READS.fq.gz -g 2800000 -i 0 --threads 20 -o flye 2>&1 | sed 's/^/[flye] /' | tee -a dragonflye.log [flye] [2022-12-05 11:47:23] INFO: Starting Flye 2.9.1-b1780 [flye] [2022-12-05 11:47:23] INFO: >>>STAGE: configure [flye] [2022-12-05 11:47:23] INFO: Configuring run [flye] [2022-12-05 11:47:29] INFO: Total read length: 432559422 [flye] [2022-12-05 11:47:29] INFO: Input genome size: 2800000 [flye] [2022-12-05 11:47:29] INFO: Estimated coverage: 154 [flye] [2022-12-05 11:47:29] INFO: Reads N50/N90: 5835 / 2000 [flye] [2022-12-05 11:47:29] INFO: Minimum overlap set to 2000 [flye] [2022-12-05 11:47:29] INFO: >>>STAGE: assembly [flye] [2022-12-05 11:47:29] INFO: Assembling disjointigs [flye] [2022-12-05 11:47:29] INFO: Reading sequences [flye] [2022-12-05 11:47:35] INFO: Counting k-mers: [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-05 11:48:22] INFO: Filling index table (1/2) [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-05 11:48:35] INFO: Filling index table (2/2) [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-05 11:48:58] INFO: Extending reads [flye] [2022-12-05 11:49:58] INFO: Overlap-based coverage: 115 [flye] [2022-12-05 11:49:58] INFO: Median overlap divergence: 0.0611082 [flye] 0% 10% 90% 100% [flye] [2022-12-05 11:51:10] INFO: Assembled 6 disjointigs [flye] [2022-12-05 11:51:10] INFO: Generating sequence [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-05 11:51:12] INFO: Filtering contained disjointigs [flye] 0% 10% 30% 50% 60% 80% 100% [flye] [2022-12-05 11:51:13] INFO: Contained seqs: 0 [flye] [2022-12-05 11:51:13] INFO: >>>STAGE: consensus [flye] [2022-12-05 11:51:13] INFO: Running Minimap2 [flye] [2022-12-05 11:51:40] INFO: Computing consensus [flye] [2022-12-05 11:53:26] INFO: Alignment error rate: 0.072399 [flye] [2022-12-05 11:53:26] INFO: >>>STAGE: repeat [flye] [2022-12-05 11:53:26] INFO: Building and resolving repeat graph [flye] [2022-12-05 11:53:26] INFO: Parsing disjointigs [flye] [2022-12-05 11:53:26] INFO: Building repeat graph [flye] 0% 10% 30% 50% 60% 80% 100% [flye] [2022-12-05 11:53:29] INFO: Median overlap divergence: 0.00331187 [flye] [2022-12-05 11:53:29] INFO: Parsing reads [flye] [2022-12-05 11:53:34] INFO: Aligning reads to the graph [flye] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [flye] [2022-12-05 11:53:50] INFO: Aligned read sequence: 385741125 / 389252928 (0.990978) [flye] [2022-12-05 11:53:50] INFO: Median overlap divergence: 0.016065 [flye] [2022-12-05 11:53:50] INFO: Mean edge coverage: 123 [flye] [2022-12-05 11:53:50] INFO: Simplifying the graph [flye] [2022-12-05 11:53:50] INFO: >>>STAGE: contigger [flye] [2022-12-05 11:53:50] INFO: Generating contigs [flye] [2022-12-05 11:53:50] INFO: Reading sequences [flye] [2022-12-05 11:53:55] INFO: Generated 6 contigs [flye] [2022-12-05 11:53:55] INFO: Added 0 scaffold connections [flye] [2022-12-05 11:53:55] INFO: >>>STAGE: finalize [flye] [2022-12-05 11:53:55] INFO: Assembly statistics: [flye] [flye] Total length: 3121116 [flye] Fragments: 6 [flye] Fragments N50: 2791405 [flye] Largest frg: 2791405 [flye] Scaffolds: 0 [flye] Mean coverage: 122 [flye] [flye] [2022-12-05 11:53:55] INFO: Final assembly: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/assembly.fasta [dragonflye] Polishing with Racon (1 rounds) [dragonflye] Running: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz 2>&1 1> flye/polish/racon/1/aligments.paf | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0571.01] collected minimizers [polishing - racon (1 of 1)] [M::mm_idx_gen::0.0632.43] sorted minimizers [polishing - racon (1 of 1)] [M::main::0.0632.43] loaded/built the index for 6 target sequence(s) [polishing - racon (1 of 1)] [M::mm_mapopt_update::0.0692.31] mid_occ = 25 [polishing - racon (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 6 [polishing - racon (1 of 1)] [M::mm_idx_stat::0.0742.22] distinct minimizers: 531318 (96.61% are singletons); average occurrences: 1.100; average spacing: 5.339; total length: 3121116 [polishing - racon (1 of 1)] [M::worker_pipeline::11.12011.61] mapped 101377 sequences [polishing - racon (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - racon (1 of 1)] [M::main] CMD: minimap2 -t 19 -x map-ont flye.fasta READS.fq.gz [polishing - racon (1 of 1)] [M::main] Real time: 11.125 sec; CPU: 129.068 sec; Peak RSS: 0.491 GB [dragonflye] Running: racon -t 20 READS.fq.gz flye/polish/racon/1/aligments.paf flye.fasta 2>&1 1> flye/polish/racon/1/consensus.fasta | sed 's/^/[polishing - racon (1 of 1)] /' | tee -a dragonflye.log [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded target sequences 0.011601 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded sequences 4.978672 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] loaded overlaps 0.090052 s [racon::Polisher::initialize] aligning overlaps [====================] 8.640065 s ] 0.737275 s [polishing - racon (1 of 1)] [racon::Polisher::initialize] transformed data into windows 0.508076 s [racon::Polisher::polish] generating consensus [====================] 47.499805 s ] 2.754352 s [polishing - racon (1 of 1)] [racon::Polisher::] total = 61.797770 s [dragonflye] Polishing with Medaka (1 rounds) [dragonflye] Running: medaka_consensus -i READS.fq.gz -d flye/polish/racon/1/consensus.fasta -o flye/polish/medaka/1 -m r941_min_sup_g507 -t 20 2>&1 | sed 's/^/[polishing - medaka (1 of 1)] /' | tee -a dragonflye.log [polishing - medaka (1 of 1)] Checking program versions [polishing - medaka (1 of 1)] This is medaka 1.7.2 [polishing - medaka (1 of 1)] Program Version Required Pass
[polishing - medaka (1 of 1)] bcftools 1.16 1.11 True
[polishing - medaka (1 of 1)] bgzip 1.16 1.11 True
[polishing - medaka (1 of 1)] minimap2 2.24 2.11 True
[polishing - medaka (1 of 1)] samtools 1.16.1 1.11 True
[polishing - medaka (1 of 1)] tabix 1.16 1.11 True
[polishing - medaka (1 of 1)] Aligning basecalls to draft [polishing - medaka (1 of 1)] Creating fai index file /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.fai [polishing - medaka (1 of 1)] Creating mmi index file /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.0941.03] collected minimizers [polishing - medaka (1 of 1)] [M::mm_idx_gen::0.1041.21] sorted minimizers [polishing - medaka (1 of 1)] [M::main::0.1211.18] loaded/built the index for 6 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 6 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.1261.17] distinct minimizers: 528528 (96.41% are singletons); average occurrences: 1.106; average spacing: 5.339; total length: 3122263 [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -I 16G -x map-ont -d /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta [polishing - medaka (1 of 1)] [M::main] Real time: 0.129 sec; CPU: 0.151 sec; Peak RSS: 0.032 GB [polishing - medaka (1 of 1)] [M::main::0.0201.02] loaded/built the index for 6 target sequence(s) [polishing - medaka (1 of 1)] [M::mm_mapopt_update::0.0251.01] mid_occ = 27 [polishing - medaka (1 of 1)] [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 6 [polishing - medaka (1 of 1)] [M::mm_idx_stat::0.0291.01] distinct minimizers: 528528 (96.41% are singletons); average occurrences: 1.106; average spacing: 5.339; total length: 3122263 [polishing - medaka (1 of 1)] [M::worker_pipeline::21.74713.36] mapped 101377 sequences [polishing - medaka (1 of 1)] [M::main] Version: 2.24-r1122 [polishing - medaka (1 of 1)] [M::main] CMD: minimap2 -x map-ont --secondary=no -L --MD -A 2 -B 4 -O 4,24 -E 2,1 -t 20 -a /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye/polish/racon/1/consensus.fasta.map-ont.mmi /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/READS.filt.fq.gz [polishing - medaka (1 of 1)] [M::main] Real time: 21.750 sec; CPU: 290.535 sec; Peak RSS: 1.760 GB [polishing - medaka (1 of 1)] [bam_sort_core] merging from 0 files and 20 in-memory blocks... [polishing - medaka (1 of 1)] Running medaka consensus [polishing - medaka (1 of 1)] [11:55:45 - Predict] Reducing threads to 2, anymore is a waste. [polishing - medaka (1 of 1)] [11:55:45 - Predict] It looks like you are running medaka without a GPU and attempted to set a high number of threads. We have scaled this down to an optimal number. If you wish to improve performance please see https://nanoporetech.github.io/medaka/installation.html#improving-parallelism. [polishing - medaka (1 of 1)] [11:55:45 - Predict] Setting tensorflow inter/intra-op threads to 2/1. [polishing - medaka (1 of 1)] [11:55:45 - Predict] Processing region(s): contig_1:0-2792966 contig_2:0-235945 contig_3:0-6180 contig_4:0-8803 contig_5:0-32438 contig_6:0-45931 [polishing - medaka (1 of 1)] [11:55:45 - Predict] Using model: /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/data/r941_min_sup_g507_model.tar.gz. [polishing - medaka (1 of 1)] [11:55:45 - Predict] Processing 8 long region(s) with batching. [polishing - medaka (1 of 1)] [11:55:46 - MdlStrTF] Model <keras.engine.sequential.Sequential object at 0x7fbd41bb0d30> [polishing - medaka (1 of 1)] [11:55:46 - MdlStrTF] loading weights from /tmp/tmpd821pglo/model/variables/variables [polishing - medaka (1 of 1)] [11:55:46 - BAMFile] Creating pool of 16 BAM file sets. [polishing - medaka (1 of 1)] [11:55:46 - Sampler] Initializing sampler for consensus of region contig_1:0-1000000. [polishing - medaka (1 of 1)] [11:55:46 - Sampler] Initializing sampler for consensus of region contig_1:999000-1999000. [polishing - medaka (1 of 1)] [11:55:46 - PWorker] Running inference for 3.1M draft bases. [polishing - medaka (1 of 1)] [11:55:48 - Feature] Processed contig_1:0.0-999999.4 (median depth 138.0) [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Took 1.66s to make features. [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Initializing sampler for consensus of region contig_1:1998000-2792966. [polishing - medaka (1 of 1)] [11:55:48 - Feature] Processed contig_1:999000.0-1998999.1 (median depth 120.0) [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Took 1.78s to make features. [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Initializing sampler for consensus of region contig_2:0-235945. [polishing - medaka (1 of 1)] [11:55:48 - Feature] Processed contig_2:0.0-235944.0 (median depth 168.0) [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Took 0.64s to make features. [polishing - medaka (1 of 1)] [11:55:48 - Sampler] Initializing sampler for consensus of region contig_3:0-6180. [polishing - medaka (1 of 1)] [11:55:49 - Feature] Processed contig_3:0.0-6179.0 (median depth 327.0) [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Took 0.24s to make features. [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Initializing sampler for consensus of region contig_4:0-8803. [polishing - medaka (1 of 1)] [11:55:49 - Feature] Processed contig_1:1998000.0-2792965.0 (median depth 138.0) [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Took 0.95s to make features. [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Initializing sampler for consensus of region contig_5:0-32438. [polishing - medaka (1 of 1)] [11:55:49 - Feature] Processed contig_4:0.0-8802.0 (median depth 273.0) [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Took 0.19s to make features. [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Initializing sampler for consensus of region contig_6:0-45931. [polishing - medaka (1 of 1)] [11:55:49 - Feature] Processed contig_5:0.0-32437.0 (median depth 291.0) [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Took 0.39s to make features. [polishing - medaka (1 of 1)] [11:55:49 - Feature] Processed contig_6:0.0-45930.0 (median depth 189.0) [polishing - medaka (1 of 1)] [11:55:49 - Sampler] Took 0.40s to make features. [polishing - medaka (1 of 1)] [11:55:58 - PWorker] Batches in cache: 7. [polishing - medaka (1 of 1)] [11:55:58 - PWorker] 14.8% Done (0.5/3.1 Mbases) in 12.1s [polishing - medaka (1 of 1)] [11:56:06 - PWorker] Batches in cache: 6. [polishing - medaka (1 of 1)] [11:56:14 - PWorker] Batches in cache: 5. [polishing - medaka (1 of 1)] [11:56:14 - PWorker] 46.2% Done (1.4/3.1 Mbases) in 28.0s [polishing - medaka (1 of 1)] [11:56:22 - PWorker] Batches in cache: 4. [polishing - medaka (1 of 1)] [11:56:30 - PWorker] Batches in cache: 3. [polishing - medaka (1 of 1)] [11:56:30 - PWorker] 76.4% Done (2.4/3.1 Mbases) in 44.0s [polishing - medaka (1 of 1)] [11:56:38 - PWorker] Batches in cache: 2. [polishing - medaka (1 of 1)] [11:56:44 - PWorker] Batches in cache: 1. [polishing - medaka (1 of 1)] [11:56:44 - PWorker] 100.0% Done (3.1/3.1 Mbases) in 57.9s [polishing - medaka (1 of 1)] [11:56:44 - PWorker] Processed 7 batches [polishing - medaka (1 of 1)] [11:56:44 - PWorker] All done, 0 remainder regions. [polishing - medaka (1 of 1)] [11:56:44 - Predict] Finished processing all regions. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] [11:56:46 - DataIndx] Loaded 1/1 (100.00%) sample files. [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] /home/gilmansiu3/miniconda3/envs/test-dragonflye/lib/python3.8/site-packages/medaka/labels.py:387: RuntimeWarning: divide by zero encountered in log10 [polishing - medaka (1 of 1)] q = -10 np.log10(err) [polishing - medaka (1 of 1)] Polished assembly written to flye/polish/medaka/1/consensus.fasta, have a nice day. [dragonflye] Using assembly file 'flye/polish/medaka/1/consensus.fasta' => 'flye.fasta' [dragonflye] Assembly is 3123154, estimated genome size was 2800000 (+11.54%) [dragonflye] It contains 6 (min=500) contigs totalling 3123154 bp. [dragonflye] Getting more stats now... [dragonflye] Running: assembly-scan --transpose contigs.fa 2>&1 | sed 's/^/[stats] /' | tee -a dragonflye.log [stats] contigs.fa total_contig 6 [stats] contigs.fa total_contig_length 3123154 [stats] contigs.fa max_contig_length 2793645 [stats] contigs.fa mean_contig_length 520525 [stats] contigs.fa median_contig_length 39190 [stats] contigs.fa min_contig_length 6179 [stats] contigs.fa n50_contig_length 2793645 [stats] contigs.fa l50_contig_count 1 [stats] contigs.fa num_contig_non_acgtn 0 [stats] contigs.fa contig_percent_a 31.01 [stats] contigs.fa contig_percent_c 18.89 [stats] contigs.fa contig_percent_g 18.84 [stats] contigs.fa contig_percent_t 31.27 [stats] contigs.fa contig_percent_n 0.00 [stats] contigs.fa contig_non_acgtn 0.00 [stats] contigs.fa contigs_greater_1m 1 [stats] contigs.fa contigs_greater_100k 2 [stats] contigs.fa contigs_greater_10k 4 [stats] contigs.fa contigs_greater_1k 6 [stats] contigs.fa percent_contigs_greater_1m 16.67 [stats] contigs.fa percent_contigs_greater_100k 33.33 [stats] contigs.fa percent_contigs_greater_10k 66.67 [stats] contigs.fa percent_contigs_greater_1k 100.00 [dragonflye] Using genome graph file 'flye/assembly_graph.gfa' => 'flye-unpolished.gfa' [dragonflye] Walltime used: 9 min 40 sec [dragonflye] Results in: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1 [dragonflye] Final assembly graph: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/flye-unpolished.gfa [dragonflye] Final assembly contigs: /mnt/data/Species-specific/CAUR/04_dragonflye/220818_VRE1/contigs.fa [dragonflye] Dragonflye is heavily based off Shovill (https://github.com/tseemann/shovill), the Illumina counterpart [dragonflye] Done. (test-dragonflye) gilmansiu3@gilmansiu3-Z490-AORUS-ULTRA:/mnt/data/Species-specific/CAUR$
Hi @llk578496
I think this all got sorted out some time in the past. PLease feel free to reopen if not!
Hello @rpetit3 ,
Thank you for developing
dragonflye
.We are trying to use
dragonflye
to perform assembly on the E. faecium isolates with the following command:dragonflye --reads 02_fastq/220818_VRE1.fastq.gz --gsize 2.8M --outdir 04_dragonflye/220818_VRE1 --cpus 20 --nanohq --model r941_min_sup_g507
However, we encounted an error in the polishing step when using medaka. Please find the error logs below:
Best regards, Eddie