Closed nick-youngblut closed 2 years ago
Installing contextvars AND zoneinfo (mamba install conda-forge::contextvars conda-forge::backports.zoneinfo
) seems to be moving the needle closer:
$ anvi-self-test --suite mini
Traceback (most recent call last):
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/bin/anvi-self-test", line 91, in <module>
from anvio.argparse import ArgumentParser
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/anvio/argparse.py", line 15, in <module>
from anvio.programs import Program
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/anvio/programs.py", line 21, in <module>
from anvio.summaryhtml import SummaryHTMLOutput
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/anvio/summaryhtml.py", line 53, in <module>
django.setup()
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/django/__init__.py", line 16, in setup
from django.urls import set_script_prefix
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/django/urls/__init__.py", line 1, in <module>
from .base import (
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/django/urls/base.py", line 8, in <module>
from .exceptions import NoReverseMatch, Resolver404
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/django/urls/exceptions.py", line 1, in <module>
from django.http import Http404
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/django/http/__init__.py", line 5, in <module>
from django.http.response import (
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/django/http/response.py", line 16, in <module>
from django.core.serializers.json import DjangoJSONEncoder
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/django/core/serializers/__init__.py", line 23, in <module>
from django.core.serializers.base import SerializerDoesNotExist
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/django/core/serializers/base.py", line 8, in <module>
from django.db import models
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/django/db/models/__init__.py", line 3, in <module>
from django.db.models.aggregates import * # NOQA
File "/tmp/global2/nyoungblut/code/dev/tmp/anvio/conda_envs/anvio/lib/python3.6/site-packages/django/db/models/aggregates.py", line 63
if (default := c.default) is None:
^
SyntaxError: invalid syntax
# Name Version Build Channel
contextvars 2.4 py_0 conda-forge
backports.zoneinfo 0.2.1 py36h8f6f2f9_4 conda-forge
I could create a github action for testing the conda install (both conda install anvio
and the longer install instructions) if you'd accept the PR.
Hey @nick-youngblut, thank you very much for the report. We gave up on the conda package for anvi'o a while ago due to tremendously long install times. So it is no surprise the conda package for v7.1
no longer works :/
The official installation instructions for anvi'o online work, but I can see the disadvantage of not having working conda packages for anvi'o :/
Well, we would accept any PR from you with pleasure!!! Thank you very much for your interest and willingness to take a stab at this. But there is something I'd like to bring to your attention first before you invest any time into this:
Just today I realized that we can no longer avoid a switch from Python 3.6 to 3.10 for anvi'o (#1911). Which will be a bit painful, and I don't know the extent of work it will require but I plan to do it ASAP. But if you think what you plan to do would be a waste of time if I were to change the installation instructions (i.e., the Python version, versions of conda and/or Python packages, etc), then I'd suggest you to wait a little longer if you wish.
Best wishes,
We gave up on the conda package for anvi'o a while ago due to tremendously long install times
This is where mamba really helps. It might be useful to encourage users to switch to mamba for the install instructions, given all of the deps.
Changing the github workflow after your update from python=3.6
to python=3.10
should be rather easy.
More generally, adding CI to the anvio repo could help you spot errors during your 3.6 => 3.10
update.
This is where mamba really helps.
Yes, but it was also not working for previous versions of anvi'o. We never were able to go after why it was not working.
More generally, adding CI to the anvio repo could help you spot errors during your 3.6 => 3.10 update.
I have no experience with CI so far, but I know it is a critical tool that can help us spot issues early on. So we would be happy for any suggestions and/or guidance if someone were to take it on and run with it.
Best wishes,
I have no experience with CI so far, but I know it is a critical tool that can help us spot issues early on. So we would be happy for any suggestions and/or guidance if someone were to take it on and run with it.
It's gotten easier with github actions versus travis-ci or circle-ci. My PR would just consist of a quickly modified version of one of my existing workflows. Maybe also a pypi publish workflow
Any reason why you have:
conda install -y -c bioconda "sqlite >=3.31.1"
conda install -y -c bioconda prodigal
conda install -y -c bioconda mcl
conda install -y -c bioconda muscle=3.8.1551
conda install -y -c bioconda hmmer
conda install -y -c bioconda diamond
conda install -y -c bioconda blast
conda install -y -c bioconda megahit
conda install -y -c bioconda spades
conda install -y -c bioconda bowtie2 tbb=2019.8
conda install -y -c bioconda bwa
conda install -y -c bioconda samtools=1.9
conda install -y -c bioconda centrifuge
conda install -y -c bioconda trimal
conda install -y -c bioconda iqtree
conda install -y -c bioconda trnascan-se
conda install -y -c bioconda r-base
conda install -y -c bioconda r-stringi
conda install -y -c bioconda r-tidyverse
conda install -y -c bioconda r-magrittr
conda install -y -c bioconda r-optparse
conda install -y -c bioconda bioconductor-qvalue
conda install -y -c bioconda fasttree
instead of just:
conda install -y -c bioconda "sqlite >=3.31.1" prodigal mcl muscle=3.8.1551 hmmer diamond blast megahit spades bowtie2 tbb=2019.8 bwa samtools=1.9 centrifuge trimal iqtree trnascan-se r-base r-stringi r-tidyverse r-magrittr r-optparse bioconductor-qvalue fasttree
# or mamba install
If any of the recipes fails to install, conda will point out the particular failed dep.
During my tests the dependency resolver was taking MUCH longer than what would be the total amount of time for individual steps. I'd be happy to be proven wrong about it.
During my tests the dependency resolver was taking MUCH longer than what would be the total amount of time for individual steps. I'd be happy to be proven wrong about it.
On my system, mamba took ~13 minutes for mamba install -y -c bioconda "sqlite >=3.31.1" prodigal mcl muscle=3.8.1551 hmmer diamond blast megahit spades bowtie2 tbb=2019.8 bwa samtools=1.9 centrifuge trimal iqtree trnascan-se r-base r-stringi r-tidyverse r-magrittr r-optparse bioconductor-qvalue fasttree
It's the downside of creating an all-in-one toolset instead of multiple modular package (e.g., tidyverse as 1 package versus tidyverse representing many individual packages). I'm guessing that modularizing anvio would probably take too much work at this point. At least mamba can help speed up the install of the long dependency list.
Still longer than individual installations with conda, but this is very promising given the added bonus of resolving a lot of headaches in one go.
PR with a github workflow for CI: https://github.com/merenlab/anvio/pull/1913
I'm also getting this error from the bioconda installation. How can I get this working? I just downloaded the Delmont 2018 genomes but they are all in CONTIGS.db and CONTIGS.h5 which I can't use. I need fasta format. I'm getting the contextvars install error but when I installed contextvars it dropped my python version down to a version that doesn't have walrus operators.
Hey @jolespin, you shouldn't install anvi'o from bioconda. I'm not sure why you went to that rabbit hole. If you follow the installation instructions on anvio.org things should work out just fine. Then anvi-migrate will migrate those old databases to the latest version, and anvi-export-contigs will give you FASTA files.
Ok awesome, I'll try it out right now! Once I get it working I'll post the conda environment here that might be useful.
Hopefully this will be helpful for someone.
mamba create -n anvio_env python=3.6
conda activate anvio_env # my mamba init is broken
mamba install -c bioconda 'sqlite>=3.31.1' prodigal mcl muscle=3.8.1551 hmmer diamond blast megahit spades bowtie2 bwa samtools=1.9 centrifuge trimal iqtree trnascan-se r-base r-stringi r-tidyverse r-magrittr r-optparse bioconductor-qvalue fasttree vmatch
Somebody can just use the yml file to recreate the environment quicker:
Call it anvio-dependencies.yml
:
name: anvio_env
channels:
- conda-forge
- bioconda
- defaults
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=2_gnu
- _r-mutex=1.0.1=anacondar_1
- _sysroot_linux-64_curr_repodata_hack=3=h69a702a_13
- binutils_impl_linux-64=2.36.1=h193b22a_2
- binutils_linux-64=2.36=hf3e587d_33
- bioconductor-qvalue=2.18.0=r36_1
- blast=2.14.0=h7d5a4b4_1
- bowtie2=2.5.1=py36hb79b6da_1
- bwa=0.7.17=he4a0461_11
- bwidget=1.9.14=ha770c72_1
- bzip2=1.0.8=h7f98852_4
- ca-certificates=2023.5.7=hbcca054_0
- cairo=1.16.0=h18b612c_1001
- centrifuge=1.0.4_beta=py36pl526he941832_2
- curl=7.68.0=hf8cf82a_0
- diamond=2.1.7=h5b5514e_0
- entrez-direct=16.2=he881be0_1
- expat=2.5.0=hcb278e6_1
- fasttree=2.1.11=h031d066_2
- fontconfig=2.14.2=h14ed4e7_0
- freetype=2.12.1=hca18f0e_1
- gcc_impl_linux-64=7.5.0=habd7529_20
- gcc_linux-64=7.5.0=h47867f9_33
- gettext=0.21.1=h27087fc_0
- gfortran_impl_linux-64=7.5.0=h56cb351_20
- gfortran_linux-64=7.5.0=h78c8a43_33
- glib=2.66.3=h58526e2_0
- graphite2=1.3.13=h58526e2_1001
- gsl=2.5=h294904e_1
- gxx_impl_linux-64=7.5.0=hd0bb8aa_20
- gxx_linux-64=7.5.0=h555fc39_33
- harfbuzz=2.4.0=h37c48d4_1
- hmmer=3.3.2=hdbdd923_4
- htslib=1.9=h244ad75_9
- icu=58.2=hf484d3e_1000
- infernal=1.1.4=h779adbc_0
- iqtree=2.2.2.3=h2202e69_2
- jpeg=9e=h0b41bf4_3
- kernel-headers_linux-64=3.10.0=h4a8ded7_13
- krb5=1.16.4=h2fd8d38_0
- ld_impl_linux-64=2.36.1=hea4e1c9_2
- lerc=4.0.0=h27087fc_0
- libblas=3.9.0=13_linux64_openblas
- libcblas=3.9.0=13_linux64_openblas
- libcurl=7.68.0=hda55be3_0
- libdeflate=1.14=h166bdaf_0
- libedit=3.1.20191231=h46ee950_2
- libexpat=2.5.0=hcb278e6_1
- libffi=3.2.1=he1b5a44_1007
- libgcc-devel_linux-64=7.5.0=hda03d7c_20
- libgcc-ng=13.1.0=he5830b7_0
- libgfortran-ng=7.5.0=h14aa051_20
- libgfortran4=7.5.0=h14aa051_20
- libglib=2.66.3=hbe7bbb4_0
- libgomp=13.1.0=he5830b7_0
- libhwloc=2.9.1=hd6dc26d_0
- libiconv=1.17=h166bdaf_0
- libidn2=2.3.4=h166bdaf_0
- liblapack=3.9.0=13_linux64_openblas
- libnsl=2.0.0=h7f98852_0
- libopenblas=0.3.18=hf726d26_0
- libpng=1.6.39=h753d276_0
- libsqlite=3.42.0=h2797004_0
- libssh2=1.10.0=haa6b8db_3
- libstdcxx-devel_linux-64=7.5.0=hb016644_20
- libstdcxx-ng=13.1.0=hfd8a6a1_0
- libtiff=4.4.0=h82bc61c_5
- libunistring=0.9.10=h7f98852_0
- libuuid=2.38.1=h0b41bf4_0
- libwebp-base=1.3.0=h0b41bf4_0
- libxcb=1.15=h0b41bf4_0
- libxml2=2.10.3=hcbfbd50_0
- libzlib=1.2.13=h166bdaf_4
- llvm-openmp=8.0.1=hc9558a2_0
- make=4.3=hd18ef5c_1
- mcl=14.137=pl5262h779adbc_6
- megahit=1.2.9=h43eeafb_4
- muscle=3.8.1551=h7d875b9_6
- ncurses=6.1=hf484d3e_1002
- openmp=8.0.1=0
- openssl=1.1.1u=hd590300_0
- pandoc=2.19.2=h32600fe_2
- pango=1.40.14=he7ab937_1005
- pcre=8.45=h9c3ff4c_0
- perl=5.26.2=h36c2ea0_1008
- perl-archive-tar=2.32=pl526_0
- perl-carp=1.38=pl526_3
- perl-common-sense=3.74=pl526_2
- perl-compress-raw-bzip2=2.087=pl526he1b5a44_0
- perl-compress-raw-zlib=2.087=pl526hc9558a2_0
- perl-exporter=5.72=pl526_1
- perl-exporter-tiny=1.002001=pl526_0
- perl-extutils-makemaker=7.36=pl526_1
- perl-io-compress=2.087=pl526he1b5a44_0
- perl-io-zlib=1.10=pl526_2
- perl-json=4.02=pl526_0
- perl-json-xs=2.34=pl526h6bb024c_3
- perl-list-moreutils=0.428=pl526_1
- perl-list-moreutils-xs=0.428=pl526_0
- perl-pathtools=3.75=pl526h14c3975_1
- perl-scalar-list-utils=1.52=pl526h516909a_0
- perl-types-serialiser=1.0=pl526_2
- perl-xsloader=0.24=pl526_0
- pip=21.3.1=pyhd8ed1ab_0
- pixman=0.38.0=h516909a_1003
- prodigal=2.6.3=h031d066_6
- pthread-stubs=0.4=h36c2ea0_1001
- python=3.6.10=h8356626_1011_cpython
- python_abi=3.6=2_cp36m
- r-askpass=1.1=r36hcfec24a_2
- r-assertthat=0.2.1=r36h6115d3f_2
- r-backports=1.2.1=r36hcfec24a_0
- r-base=3.6.1=h8900bf8_2
- r-base64enc=0.1_3=r36hcfec24a_1004
- r-blob=1.2.1=r36h6115d3f_1
- r-brio=1.1.2=r36hcfec24a_0
- r-broom=0.7.6=r36hc72bb7e_0
- r-callr=3.7.0=r36hc72bb7e_0
- r-cellranger=1.1.0=r36h6115d3f_1003
- r-cli=2.5.0=r36hc72bb7e_0
- r-clipr=0.7.1=r36h142f84f_0
- r-colorspace=2.0_1=r36hcfec24a_0
- r-cpp11=0.2.7=r36hc72bb7e_0
- r-crayon=1.4.1=r36hc72bb7e_0
- r-curl=4.3.1=r36hcfec24a_0
- r-data.table=1.14.0=r36hcfec24a_0
- r-dbi=1.1.1=r36hc72bb7e_0
- r-dbplyr=2.1.1=r36hc72bb7e_0
- r-desc=1.3.0=r36hc72bb7e_0
- r-diffobj=0.3.4=r36hcfec24a_0
- r-digest=0.6.27=r36h03ef668_0
- r-dplyr=1.0.6=r36h03ef668_1
- r-dtplyr=1.1.0=r36hc72bb7e_0
- r-ellipsis=0.3.2=r36hcfec24a_0
- r-evaluate=0.14=r36h6115d3f_2
- r-fansi=0.4.2=r36hcfec24a_0
- r-farver=2.1.0=r36h03ef668_0
- r-forcats=0.5.1=r36hc72bb7e_0
- r-fs=1.5.0=r36h0357c0b_0
- r-gargle=1.1.0=r36hc72bb7e_0
- r-generics=0.1.0=r36hc72bb7e_0
- r-getopt=1.20.3=r36_2
- r-ggplot2=3.3.3=r36hc72bb7e_0
- r-glue=1.4.2=r36hcfec24a_0
- r-googledrive=1.0.1=r36h6115d3f_1
- r-googlesheets4=0.3.0=r36hc72bb7e_0
- r-gtable=0.3.0=r36h6115d3f_3
- r-haven=2.4.1=r36h2713e49_0
- r-highr=0.9=r36hc72bb7e_0
- r-hms=1.1.0=r36hc72bb7e_0
- r-htmltools=0.5.1.1=r36h03ef668_0
- r-httr=1.4.2=r36h6115d3f_0
- r-ids=1.0.1=r36h6115d3f_1
- r-isoband=0.2.4=r36h03ef668_0
- r-jsonlite=1.7.2=r36hcfec24a_0
- r-knitr=1.33=r36hc72bb7e_0
- r-labeling=0.4.2=r36h142f84f_0
- r-lattice=0.20_44=r36hcfec24a_0
- r-lifecycle=1.0.0=r36hc72bb7e_0
- r-lubridate=1.7.10=r36h03ef668_0
- r-magrittr=2.0.1=r36hcfec24a_1
- r-markdown=1.1=r36hcfec24a_1
- r-mass=7.3_54=r36hcfec24a_0
- r-matrix=1.3_3=r36he454529_0
- r-mgcv=1.8_35=r36he454529_0
- r-mime=0.10=r36hcfec24a_0
- r-modelr=0.1.8=r36h6115d3f_0
- r-munsell=0.5.0=r36h6115d3f_1003
- r-nlme=3.1_150=r36h31ca83e_0
- r-openssl=1.4.4=r36he36bf35_0
- r-optparse=1.6.6=r36h6115d3f_1
- r-pillar=1.6.1=r36hc72bb7e_0
- r-pkgconfig=2.0.3=r36h6115d3f_1
- r-pkgload=1.2.1=r36h03ef668_0
- r-plyr=1.8.6=r36h0357c0b_1
- r-praise=1.0.0=r36h6115d3f_1004
- r-prettyunits=1.1.1=r36h6115d3f_1
- r-processx=3.5.2=r36hcfec24a_0
- r-progress=1.2.2=r36h6115d3f_2
- r-ps=1.6.0=r36hcfec24a_0
- r-purrr=0.3.4=r36hcfec24a_1
- r-r6=2.5.0=r36hc72bb7e_0
- r-rappdirs=0.3.3=r36hcfec24a_0
- r-rcolorbrewer=1.1_2=r36h6115d3f_1003
- r-rcpp=1.0.6=r36h03ef668_0
- r-readr=1.4.0=r36h1b71b39_0
- r-readxl=1.3.1=r36hde08347_4
- r-rematch=1.0.1=r36h6115d3f_1003
- r-rematch2=2.1.2=r36h6115d3f_1
- r-reprex=2.0.0=r36hc72bb7e_0
- r-reshape2=1.4.4=r36h0357c0b_1
- r-rlang=0.4.11=r36hcfec24a_0
- r-rmarkdown=2.8=r36hc72bb7e_0
- r-rprojroot=2.0.2=r36hc72bb7e_0
- r-rstudioapi=0.13=r36hc72bb7e_0
- r-rvest=1.0.0=r36hc72bb7e_0
- r-scales=1.1.1=r36h6115d3f_0
- r-selectr=0.4_2=r36h6115d3f_1
- r-stringi=1.4.3=r36h0357c0b_2
- r-stringr=1.4.0=r36h6115d3f_2
- r-sys=3.4=r36hcfec24a_0
- r-testthat=3.0.2=r36h03ef668_0
- r-tibble=3.1.2=r36hcfec24a_0
- r-tidyr=1.1.3=r36h03ef668_0
- r-tidyselect=1.1.1=r36hc72bb7e_0
- r-tidyverse=1.3.1=r36hc72bb7e_0
- r-tinytex=0.31=r36hc72bb7e_0
- r-utf8=1.2.1=r36hcfec24a_0
- r-uuid=0.1_4=r36hcdcec82_1
- r-vctrs=0.3.8=r36hcfec24a_1
- r-viridislite=0.4.0=r36hc72bb7e_0
- r-waldo=0.2.5=r36hc72bb7e_0
- r-withr=2.4.2=r36hc72bb7e_0
- r-xfun=0.23=r36hcfec24a_0
- r-xml2=1.3.2=r36h0357c0b_1
- r-yaml=2.2.1=r36hcfec24a_1
- readline=8.0=h46ee950_1
- samtools=1.9=h10a08f8_12
- sed=4.8=he412f7d_0
- setuptools=58.0.4=py36h5fab9bb_2
- spades=3.15.5=h95f258a_1
- sqlite=3.32.3=hcee41ef_1
- sysroot_linux-64=2.17=h4a8ded7_13
- tbb=2021.9.0=hf52228f_0
- tk=8.6.12=h27826a3_0
- tktable=2.10=hb7b940f_3
- trimal=1.4.1=h4ac6f70_8
- trnascan-se=2.0.9=pl5262h779adbc_1
- vmatch=2.3.0=hec16e2b_5
- wget=1.20.3=ha56f1ee_1
- wheel=0.37.1=pyhd8ed1ab_0
- xorg-kbproto=1.0.7=h7f98852_1002
- xorg-libice=1.1.1=hd590300_0
- xorg-libsm=1.2.4=h7391055_0
- xorg-libx11=1.8.4=h8ee46fc_1
- xorg-libxau=1.0.11=hd590300_0
- xorg-libxdmcp=1.1.3=h7f98852_0
- xorg-libxext=1.3.4=h0b41bf4_2
- xorg-libxrender=0.9.10=h7f98852_1003
- xorg-renderproto=0.11.1=h7f98852_1002
- xorg-xextproto=7.3.0=h0b41bf4_1003
- xorg-xproto=7.0.31=h7f98852_1007
- xz=5.2.10=h5eee18b_1
- zlib=1.2.13=h166bdaf_4
- zstd=1.5.2=h3eb15da_6
mamba env create -n anvio_env -f anvio-dependencies.yml
curl -L https://github.com/merenlab/anvio/releases/download/v7.1/anvio-7.1.tar.gz > --output anvio-7.1.tar.gz
pip install anvio-7.1.tar.gz
We tried this solution before, @jolespin. Environment YAML files are not a consistently good option for installation as a lot of people with different operating systems or even different versions of the same operating system run into problems quickly :(
Oh bummer. Yea conda is kind of a black box sometimes. I've had the issues happen a couple of times installing QIIME2 but on one of the forums I was advised to run conda clean and it actually did solve the issue. I'm sure there's lots of edge cases.
Mamba is faster than the YAML option. Setting up the environment literally takes a minute or two on a laptop since it installs all packages in parallel. Sorry mamba is broken on your system and good luck with the rest of your work :)
Luckily it's only mamba init/activate that doesn't work but all other mamba processes like create, install and uninstall work perfectly. Major increase in speed.
I wanted to add two more cents on the subject: conda is very popular and mamba is lightning fast at resolving environments. So, after playing a bit with the versions, I came up with the following environment definition that appears to pass anvi-self-test
from bioconda::anvio=7
:
channels:
- conda-forge
- bioconda
dependencies:
- anvio=7
- backports.zoneinfo=0.2.1
- contextvars=2.4
- asgiref=3.4.0
- django=3.2.20
- mistune=0.8.4
- pip
- pip:
- scikit-learn==0.24.0
mamba env create -p anvio_env -f anvio.yaml
Resolving the environment takes ~25 sec. scikit-learn
is updated from 0.20.3
to 0.24.0
with pipy
.
Short description of the problem
Conda install (
anvio 7 hdfd78af_1 bionconda
) on Ubuntu 18.04.6 produces the following error:Other commands (eg.,
anvi-migrate
) generate the same error.django is already installed in my conda env:
django 4.0 pyhd8ed1ab_0 conda-forge
Creation of the conda env:
mamba create -n anvio_env bioconda::anvio
anvi'o version
System info
Ubuntu 18.04.6
Conda env