search

nextstrain / bioconda-recipes

Fork for maintaining Nextstrain-related recipes and submitting PRs upstream.
0 stars 0 forks source link

Augur cannot be installed natively for ARM64 on Bioconda even though it is labeled "noarch" #3

Closed victorlin closed 2 months ago

victorlin commented 1 year ago

Background

Augur's Bioconda recipe page shows noarch, presumably because it is a pure-Python package that can be run natively on any architecture. In practice, this can be useful for running computationally expensive pure-Python Augur subcommands such as augur filter.

However, installing from Bioconda in an ARM64 context is currently impossible (I thought there might be a chance with --no-deps but it didn't work for me). The reason is because the recipe defines the dependencies explicitly, and at least one dependency is not available for ARM64 and the single conda install command can only search resolve packages under a single architecture. This forces all Augur subcommands to use emulation when only certain dependencies require emulation.

Possible solutions

1. Remove dependencies from Augur's Bioconda recipe

Since these dependencies are only used in a small part of Augur, I'd like to think of them as optional dependencies that can be installed independently. In other words, one would install them separately, not necessarily through Conda, if they want to use those features. This seems to be what the current Augur-specific installation page implies.

This is also how it's done for building the Docker image (1, 2) and anyone that is using Augur via pip install.

Proposed changes:

  1. Remove these lines from the Bioconda recipe.
  2. Add them to conda-base's recipe.yaml.
  3. Update ambient runtime setup's Conda install command to install these packages.

2. Bundle required dependencies into Augur's wheels

See https://github.com/nextstrain/bioconda-recipes/issues/3#issuecomment-1546143203.

3. Wait for all dependencies to be available for ARM64

See https://github.com/nextstrain/bioconda-recipes/issues/3#issuecomment-1544768125.

jameshadfield commented 1 year ago

This is presumably the same for our conda runtime, given that it installs augur?

Since these dependencies are only used in a small part of Augur, I'd like to think of them as optional dependencies that can be installed independently.

Yes and no. Augur needs a tree builder for most workflows 😉 But there are many tools that are workflow specific, and this list will grow. At some point we need to move towards workflows being able to specify these rather than including them within augur (or conda-base). (Something to keep in mind, not necessarily solve right now.)

tsibley commented 1 year ago

This seems reasonable to me given our current directions with runtimes and project packaging. Note that it will be a breaking change for anyone used to installing Augur with Conda and finding that they now need to also install some other stuff they never had to before.

Note also that we didn't originally define these deps in the Conda packaging of Augur: they were in the original third-party packaging.

victorlin commented 1 year ago

Note that it will be a breaking change for anyone used to installing Augur with Conda and finding that they now need to also install some other stuff they never had to before.

This can be done tandem with an Augur release and mentioned in the changelog. My thinking is that it would be a "note" to add after usual changelog entries without effect on semver. Example:

Note: Starting with this version, installing Augur from Conda will not automatically install fasttree, iqtree, mafft, raxml, and vcftools. Install them separately if you would like to use subcommands that are dependent on them. This increases flexibility for installation of Augur and the other software tools. (nextstrain/bioconda-recipes#3)

I'd also do this after https://github.com/nextstrain/docs.nextstrain.org/pull/157 is merged and create another PR to update the FAQ section there.

victorlin commented 1 year ago

This is presumably the same for our conda runtime, given that it installs augur?

If you're talking about the inability to use ARM64, yes. An inherent problem with Conda is that, given my limited trials, it doesn't play well when you try to install packages from different architectures into the same environment. This is because most packages (such as Augur currently) define dependencies, and dependency resolution is limited to one architecture. This contrasts with the Docker image that we build, where we have the option to build/install packages individually for the native architecture.

Because of this, the Conda runtime uses an Intel-based Miniconda installation for all Macs regardless of processor architecture (src).

sgoggins commented 1 year ago

Anaconda is rife with issues for many Python projects. We don't recommend using Anaconda. This is in our documentation.

victorlin commented 1 year ago

@sgoggins I think there's some confusion here – we're maintainers of nextstrain/augur and it looks like you're working on a tool of the same name.

huddlej commented 1 year ago

at least one dependency is not available for ARM64

Which dependencies do not have ARM64 support? Is that information published somewhere explicitly? I can't tell from the recipe meta YAML files for these packages alone...

victorlin commented 1 year ago

@huddlej you can tell from each package's Bioconda page. For example, MAFFT:

mafft-available-installers

victorlin commented 1 year ago

I just looked and none of fasttree, iqtree, mafft, raxml, or vcftools are Conda-installable for ARM64. This might be due to https://github.com/bioconda/bioconda-recipes/issues/23454.

huddlej commented 1 year ago

Thanks, @victorlin! I noticed the badges but never saw one with ARM64 specified. That Bioconda issue explains why! :D

Since all of Bioconda doesn't support ARM64, that makes this outcome less desirable for users:

In other words, one would install them separately, not necessarily through Conda, if they want to use those features.

As @jameshadfield pointed out, the features users would be missing include an aligner and tree builder which are pretty key to most Augur workflows. I would really prefer to not have to install/upgrade IQ-TREE, mafft, etc. manually and outside of my Conda environments.

Because of this, the Conda runtime uses an Intel-based Miniconda installation for all Macs regardless of processor architecture.

Instead of removing the architecture-specific dependencies from the Bioconda recipe, could we instead recommend the same approach as above of using the Intel-based Miniconda on Mac regardless of actual architecture? This is how I have Augur installed "ambiently" now. Even though this approach slows down workflows because of emulation, the other non-Docker alternative is the managed Conda environment which will also be equally slow because of emulation.

It also seems like Bioconda will eventually support ARM64, so we could eventually also recommend using the ARM64-based Miniconda in the future...

victorlin commented 1 year ago

@huddlej I see your point. I was thinking of conda install as serving similar purposes to pip install when the average Conda user probably thinks otherwise.

In other words, the average user might expect pip install to manage Python dependencies (as it does), while they might expect conda install to manage ~all dependencies (which Augur's Bioconda recipe currently does). With this perspective, I'd opt to keep things as-is.

That said, I think the deeper problem is that the average Apple silicon user is unaware of the emulation subtleties (because macOS hides them very well), meaning they aren't aware of potentially faster options. I think the best we can do here is:

  1. Continue work in the Docker runtime to build more tools from source, allowing them to be run natively.
  2. Update our installation docs to better educate Apple silicon users so they can understand how emulation would impact their workflows in various runtimes.
tsibley commented 1 year ago

Hmm. I'm confused. Maybe I'm missing something?

I would really prefer to not have to install/upgrade IQ-TREE, mafft, etc. manually and outside of my Conda environments.

Why would this change force you to install them outside of your Conda environments? It wouldn't prevent you from doing what you do now (namely, using "Intel-based Miniconda on Mac regardless of actual architecture").

It would force you to manually install/upgrade them separately from Augur itself.

huddlej commented 1 year ago

Yeah, I see how that was confusing; I'm currently a Linux, Mac OS X Intel, and ARM64 user of Augur, so here's how I'm thinking about the experience:

If I'm a new/current Linux or Mac OS X Intel user, I would have to manually install/upgrade Augur and its compiled dependencies separately.

If I'm a new/current Mac OS X ARM64 user, I would have to manually install/upgrade Augur in a Conda environment and then manually install/upgrade the dependencies outside of the Conda environment with Homebrew, etc.

For all architectures, this setup means I need to know that the additional compiled dependencies have to be installed separately and I need to know what those dependencies are and which versions to install. When I install Augur from Conda, I don't necessarily know that it can't/won't install the required compiled dependencies for me unless I'm following the docs or I try to run augur align or augur tree and it crashes.

For the Linux and Max OS X Intel users, the compiled dependencies would at least still be in a Conda environment. The new setup would be a downgrade of the current user experience where I can just install Augur on both of those systems and all dependencies are managed for me.

For the ARM64 users, their dependency management is much more complicated and loses the benefits of self-managed Conda environments. If we propose in the docs that ARM64 users install Augur with ARM64 Conda, they would not think they have an option to avoid this complexity.

The experience for the ARM64 user right now is not good at all, since they can't install Augur with Conda at all and don't get any explanation from Conda about why not. The proposed solution of removing the compiled dependencies from the Bioconda Augur recipe marginally improves the situation by allowing them to install an incomplete installation of Augur. This comes at the cost of a poorer user experience for Linux and Mac OS X Intel users (everyone gets an incomplete installation of Augur).

It seems like there are two other alternate installation paths for ARM64 users that don't degrade the experience for non-ARM64 users:

  1. Install Augur with Mac OS X Intel Conda (like I'm personally doing, like we do for conda-base environment) and get all dependencies managed for them at the cost of degraded performance for alignment and tree building.
  2. Install Augur with pip and manually install compiled dependencies built for ARM64 using Homebrew, etc. at the cost of increased dependency management but with the benefit of better performance of those tools.

When Bioconda finally supports ARM64 builds, the problem for ARM64 users goes away and Linux and OS X Intel users don't notice the difference.

tsibley commented 1 year ago

Ok, I think I see your point now: Removing MAFFT, IQ-TREE, etc. deps from the Conda packaging results in a degraded experience for anyone not in an arm64 Conda env, and that isn't made worth it by the gained ability to get an incomplete (and thus not super useful) Augur install in an arm64 Conda env.

That makes sense to me.

It does make me think that instead of making the Conda package like the Python package and removing these deps, we could actually make the Python package more like the Conda package and bundle fasttree, iqtree, mafft, raxml, or vcftools into the wheels.

mfansler commented 1 year ago

Mixing Channel Subdirectories

Theoretically, the conda/mamba CLIs are fully capable of solving with mixed channel subdirectories (e.g., architectures). Whether macOS will seamlessly handle the interoperability in the resulting environment is another question. Unfortunately, my M1 is packed for moving, so I can't test this right now, but maybe one of you might want to try something like:

mamba create -n foo --override-channels -c conda-forge -c conda-forge/osx-64 -c bioconda/osx-64 -c -c bioconda/noarch augur

This assumes:

If the system base is osx-64 (but still on Apple Silicon), the meaning of straight conda-forge is different, and so would need something like:

mamba create -n foo --override-channels -c conda-forge/osx-arm64 -c conda-forge -c bioconda augur

Note that conda-forge without a subdirectory always implicitly pulls in the noarch subdirectory.

Probably better would be to simply spell out everything explicitly in a YAML:

augur-arm64.yaml

channels:
  - conda-forge/osx-arm64  # prefer native
  - conda-forge/noarch     # or noarch
  - conda-forge/osx-64     # otherwise, emulate
  - bioconda/noarch
  - bioconda/osx-64
  - nodefaults             # equivalent of "--override-channels"
dependencies:
  - augur
mamba env create -n foo -f augur-arm64.yaml

I've never really played with this, so suggest it only for experimentation.

victorlin commented 1 year ago

@mfansler interesting idea! I just tried it on my M1.

The environment creation was successful, and I can see that packages were pulled from a mix of the arm64 and osx-64 channels.

Output ``` Package Version Build Channel Size ─────────────────────────────────────────────────────────────────────────────────────────── Install: ─────────────────────────────────────────────────────────────────────────────────────────── + attrs 23.1.0 pyh71513ae_1 conda-forge/noarch Cached + augur 22.3.0 pyhdfd78af_0 bioconda/noarch 188kB + bcbio-gff 0.7.0 pyh7cba7a3_0 bioconda/noarch 22kB + biopython 1.81 py310h8e9501a_0 conda-forge/osx-arm64 3MB + brotli 1.0.9 h1a8c8d9_9 conda-forge/osx-arm64 20kB + brotli-bin 1.0.9 h1a8c8d9_9 conda-forge/osx-arm64 18kB + brotli-python 1.0.9 py310h0f1eb42_9 conda-forge/osx-arm64 326kB + bx-python 0.10.0 py310h260c36f_0 bioconda/osx-64 1MB + bzip2 1.0.8 h3422bc3_4 conda-forge/osx-arm64 Cached + ca-certificates 2023.7.22 hf0a4a13_0 conda-forge/osx-arm64 150kB + certifi 2023.7.22 pyhd8ed1ab_0 conda-forge/noarch 154kB + cffi 1.15.1 py310h2399d43_3 conda-forge/osx-arm64 219kB + charset-normalizer 3.2.0 pyhd8ed1ab_0 conda-forge/noarch 46kB + contourpy 1.1.0 py310h38f39d4_0 conda-forge/osx-arm64 216kB + cvxopt 1.3.2 py310hadc6ad6_0 conda-forge/osx-arm64 458kB + cycler 0.11.0 pyhd8ed1ab_0 conda-forge/noarch Cached + dsdp 5.8 h9397a75_1203 conda-forge/osx-arm64 Cached + fasttree 2.1.11 he0842df_2 bioconda/osx-64 247kB + fftw 3.3.10 nompi_h4fa670e_108 conda-forge/osx-64 2MB + fonttools 4.42.0 py310h2aa6e3c_0 conda-forge/osx-arm64 2MB + freetype 2.12.1 h3f81eb7_1 conda-forge/osx-64 Cached + glpk 5.0 h3cb5acd_0 conda-forge/osx-64 Cached + gmp 6.2.1 h2e338ed_0 conda-forge/osx-64 Cached + gsl 2.7 h93259b0_0 conda-forge/osx-64 Cached + idna 3.4 pyhd8ed1ab_0 conda-forge/noarch Cached + importlib-metadata 6.8.0 pyha770c72_0 conda-forge/noarch 26kB + iqtree 2.2.3 hd98611f_0 bioconda/osx-64 8MB + isa-l 2.30.0 h0d85af4_4 conda-forge/osx-64 Cached + isodate 0.6.1 pyhd8ed1ab_0 conda-forge/noarch Cached + jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge/noarch Cached + kiwisolver 1.4.4 py310ha23aa8a_1 conda-forge/osx-64 Cached + lcms2 2.15 hd835a16_1 conda-forge/osx-arm64 206kB + lerc 4.0.0 h9a09cb3_0 conda-forge/osx-arm64 Cached + libblas 3.9.0 17_osxarm64_openblas conda-forge/osx-arm64 Cached + libbrotlicommon 1.0.9 h1a8c8d9_9 conda-forge/osx-arm64 70kB + libbrotlidec 1.0.9 h1a8c8d9_9 conda-forge/osx-arm64 29kB + libbrotlienc 1.0.9 h1a8c8d9_9 conda-forge/osx-arm64 263kB + libcblas 3.9.0 17_osxarm64_openblas conda-forge/osx-arm64 Cached + libcxx 16.0.6 h4653b0c_0 conda-forge/osx-arm64 Cached + libdeflate 1.18 hac1461d_0 conda-forge/osx-64 Cached + libffi 3.4.2 h3422bc3_5 conda-forge/osx-arm64 Cached + libgfortran 5.0.0 12_2_0_hd922786_32 conda-forge/osx-arm64 162kB + libgfortran5 12.2.0 h0eea778_32 conda-forge/osx-arm64 1MB + libjpeg-turbo 2.1.5.1 h1a8c8d9_0 conda-forge/osx-arm64 Cached + liblapack 3.9.0 17_osxarm64_openblas conda-forge/osx-arm64 Cached + libopenblas 0.3.23 openmp_h429af6e_0 conda-forge/osx-64 6MB + libpng 1.6.39 ha978bb4_0 conda-forge/osx-64 Cached + libsqlite 3.42.0 hb31c410_0 conda-forge/osx-arm64 Cached + libtiff 4.5.1 h23a1a89_0 conda-forge/osx-arm64 365kB + libwebp-base 1.3.1 hb547adb_0 conda-forge/osx-arm64 274kB + libxcb 1.15 hb7f2c08_0 conda-forge/osx-64 314kB + libzlib 1.2.13 h8a1eda9_5 conda-forge/osx-64 Cached + llvm-openmp 16.0.6 h1c12783_0 conda-forge/osx-arm64 Cached + mafft 7.520 h2413b67_2 bioconda/osx-64 9MB + matplotlib-base 3.7.2 py310h475a17b_0 conda-forge/osx-64 7MB + metis 5.1.1 h13dd4ca_0 conda-forge/osx-arm64 4MB + mpfr 4.2.0 h4f9bd69_0 conda-forge/osx-64 Cached + munkres 1.1.4 pyh9f0ad1d_0 conda-forge/noarch Cached + ncurses 6.4 h7ea286d_0 conda-forge/osx-arm64 Cached + networkx 2.8.8 pyhd8ed1ab_0 conda-forge/noarch Cached + numpy 1.25.2 py310h7451ae0_0 conda-forge/osx-64 6MB + openjpeg 2.5.0 hbc2ba62_2 conda-forge/osx-arm64 307kB + openssl 3.1.2 h8a1eda9_0 conda-forge/osx-64 2MB + packaging 23.1 pyhd8ed1ab_0 conda-forge/noarch Cached + pandas 1.5.3 py310h2b830bf_1 conda-forge/osx-arm64 11MB + pbzip2 1.1.13 h9d27c22_1 conda-forge/osx-64 Cached + perl 5.32.1 4_hf2054a2_perl5 conda-forge/osx-arm64 14MB + pigz 2.6 h5dbffcc_0 conda-forge/osx-64 Cached + pillow 10.0.0 py310h60ecbdf_0 conda-forge/osx-arm64 47MB + pip 23.2.1 pyhd8ed1ab_0 conda-forge/noarch Cached + platformdirs 3.10.0 pyhd8ed1ab_0 conda-forge/noarch 20kB + pooch 1.7.0 pyha770c72_3 conda-forge/noarch Cached + pthread-stubs 0.4 hc929b4f_1001 conda-forge/osx-64 Cached + pycparser 2.21 pyhd8ed1ab_0 conda-forge/noarch Cached + pyfastx 0.8.4 py310hdb29d00_1 bioconda/osx-64 Cached + pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge/noarch Cached + pyrsistent 0.19.3 py310h8e9501a_0 conda-forge/osx-arm64 95kB + pysocks 1.7.1 pyha2e5f31_6 conda-forge/noarch Cached + python 3.10.12 had23ca6_0_cpython conda-forge/osx-64 13MB + python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge/noarch Cached + python-isal 1.2.0 py310h6729b98_0 conda-forge/osx-64 47kB + python_abi 3.10 3_cp310 conda-forge/osx-64 Cached + pytz 2023.3 pyhd8ed1ab_0 conda-forge/noarch Cached + raxml 8.2.12 h2413b67_6 bioconda/osx-64 3MB + readline 8.2 h9e318b2_1 conda-forge/osx-64 Cached + requests 2.31.0 pyhd8ed1ab_0 conda-forge/noarch 57kB + scipy 1.11.1 py310h0975f3d_0 conda-forge/osx-arm64 14MB + setuptools 68.0.0 pyhd8ed1ab_0 conda-forge/noarch Cached + six 1.16.0 pyh6c4a22f_0 conda-forge/noarch Cached + suitesparse 5.10.1 h7aff33d_1 conda-forge/osx-64 Cached + tbb 2021.10.0 h1c7c39f_0 conda-forge/osx-64 159kB + tk 8.6.12 h5dbffcc_0 conda-forge/osx-64 Cached + treetime 0.10.1 pyh7cba7a3_0 bioconda/noarch 117kB + typing-extensions 4.7.1 hd8ed1ab_0 conda-forge/noarch 10kB + typing_extensions 4.7.1 pyha770c72_0 conda-forge/noarch 36kB + tzdata 2023c h71feb2d_0 conda-forge/noarch Cached + unicodedata2 15.0.0 py310h90acd4f_0 conda-forge/osx-64 Cached + urllib3 2.0.4 pyhd8ed1ab_0 conda-forge/noarch 98kB + vcftools 0.1.16 pl5321hdf58011_9 bioconda/osx-64 486kB + wheel 0.41.1 pyhd8ed1ab_0 conda-forge/noarch 57kB + xopen 1.7.0 py310h2ec42d9_2 conda-forge/osx-64 Cached + xorg-libxau 1.0.11 hb547adb_0 conda-forge/osx-arm64 14kB + xorg-libxdmcp 1.1.3 h35c211d_0 conda-forge/osx-64 Cached + xz 5.2.6 h775f41a_0 conda-forge/osx-64 Cached + zipp 3.16.2 pyhd8ed1ab_0 conda-forge/noarch 19kB + zlib 1.2.13 h8a1eda9_5 conda-forge/osx-64 91kB + zstandard 0.19.0 py310h07fcf8c_2 conda-forge/osx-arm64 322kB + zstd 1.5.2 h4f39d0f_7 conda-forge/osx-arm64 317kB ```

However, upon running augur --help, there was an unsurprising architecture compatibility error:

Original error was: dlopen(/opt/homebrew/Caskroom/miniconda/base/envs/tmp/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so, 0x0002): Library not loaded: '@rpath/libgfortran.5.dylib'
  Referenced from: '/opt/homebrew/Caskroom/miniconda/base/envs/tmp/lib/libopenblasp-r0.3.23.dylib'
  Reason: tried: '/opt/homebrew/Caskroom/miniconda/base/envs/tmp/lib/libgfortran.5.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64))), '/opt/homebrew/Caskroom/miniconda/base/envs/tmp/lib/python3.10/site-packages/numpy/core/../../../../libgfortran.5.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64))), '/opt/homebrew/Caskroom/miniconda/base/envs/tmp/lib/python3.10/site-packages/numpy/core/../../../../libgfortran.5.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64))), '/opt/homebrew/Caskroom/miniconda/base/envs/tmp/bin/../lib/libgfortran.5.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64))), '/opt/homebrew/Caskroom/miniconda/base/envs/tmp/bin/../lib/libgfortran.5.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64))), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file)

From the detailed logs above, the obvious reason is that numpy was installed from conda-forge/osx-64 while Augur is native. In an attempt to install numpy from conda-forge/osx-arm64 instead since it is available, I tried (1) setting the channel_priority config to strict and (2) explicitly specifying conda-forge/osx-arm64::numpy in the YAML, but none of those seemed to work. 😕

Even if this were to work, it might be unnecessary hassle for users who are just trying to mamba install augur.

Thank you for the suggestion though, the capability of conda to solving with mixed archs is good to know!

mfansler commented 1 year ago

Thanks for reporting back the results of your testing! Yes, I wasn't too hopeful and suspected that while Rosetta could handle launching new processes in emulation, it might not cover dynamic library callouts. Particularly problematic would be when two packages of different architectures both link against something like BLAS. The one would have references for the osx-arm64 BLAS and the other for osx-64 BLAS. Since Conda can't install both versions, I'd guess this could result in missing symbols references. Anyway, I suppose it was worth the try!

I do find the numpy result odd - the solver should have prioritized osx-arm64 and only fallback to osx-64 only when otherwise unavailable. Channel priority of "flexible" should be sufficient.

corneliusroemer commented 11 months ago

ARM builds are coming, first linux aarch64, but maybe also soon M1: https://github.com/bioconda/bioconda-docs/pull/16

What I do is a little hacky but it works well for me:

victorlin commented 2 months ago

Augur is now natively installable on osx-arm64, largely thanks to @corneliusroemer's work in https://github.com/nextstrain/conda-base/issues/77. I just tested locally:

micromamba install -c conda-forge -c bioconda augur ``` bioconda/osx-arm64 45.5kB @ 342.0kB/s 0.1s bioconda/noarch 5.2MB @ 5.8MB/s 0.9s conda-forge/noarch 16.3MB @ 14.9MB/s 1.1s conda-forge/osx-arm64 11.5MB @ 8.0MB/s 1.4s Transaction Prefix: /Users/victor/.micromamba/envs/tmp Updating specs: - augur Package Version Build Channel Size ────────────────────────────────────────────────────────────────────────────────── Install: ────────────────────────────────────────────────────────────────────────────────── + raxml 8.2.13 h8316a8f_2 bioconda 385kB + metis 5.1.0 h13dd4ca_1007 conda-forge Cached + libcxx 18.1.8 h3ed4263_7 conda-forge 437kB + xz 5.2.6 h57fd34a_0 conda-forge Cached + libexpat 2.6.2 hebf3989_0 conda-forge 64kB + python_abi 3.11 5_cp311 conda-forge 6kB + ncurses 6.5 h7bae524_1 conda-forge 802kB + bzip2 1.0.8 h99b78c6_7 conda-forge Cached + libffi 3.4.2 h3422bc3_5 conda-forge Cached + libzlib 1.3.1 hfb2fe0b_1 conda-forge Cached + ca-certificates 2024.8.30 hf0a4a13_0 conda-forge 158kB + libiconv 1.17 h0d3ecfb_2 conda-forge Cached + icu 75.1 hfee45f7_0 conda-forge Cached + llvm-openmp 18.1.8 hde57baf_1 conda-forge Cached + perl 5.32.1 7_h4614cfb_perl5 conda-forge 14MB + isa-l 2.31.0 h99b78c6_2 conda-forge 125kB + libwebp-base 1.4.0 h93a5062_0 conda-forge 288kB + libjpeg-turbo 3.0.0 hb547adb_1 conda-forge 548kB + libdeflate 1.21 h99b78c6_0 conda-forge 55kB + libbrotlicommon 1.1.0 hd74edd7_2 conda-forge 68kB + pthread-stubs 0.4 h27ca646_1001 conda-forge 6kB + xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge 18kB + xorg-libxau 1.0.11 hb547adb_0 conda-forge 14kB + lerc 4.0.0 h9a09cb3_0 conda-forge 216kB + qhull 2020.2 h420ef59_5 conda-forge 516kB + zlib-ng 2.2.1 h00cdb27_0 conda-forge 90kB + gmp 6.3.0 h7bae524_2 conda-forge Cached + libasprintf 0.22.5 h8414b35_3 conda-forge 41kB + readline 8.2 h92ec313_1 conda-forge Cached + pbzip2 1.1.13 h8e2fd4a_2 conda-forge 40kB + libpng 1.6.43 h091b4b1_0 conda-forge 264kB + zstd 1.5.6 hb46c0d2_0 conda-forge 405kB + pigz 2.8 h091b4b1_0 conda-forge 66kB + tk 8.6.13 h5083fa2_1 conda-forge Cached + libsqlite 3.46.1 hc14010f_0 conda-forge 830kB + zlib 1.3.1 hfb2fe0b_1 conda-forge Cached + openssl 3.3.2 h8359307_0 conda-forge 3MB + libintl 0.22.5 h8414b35_3 conda-forge 81kB + libxml2 2.12.7 h01dff8b_4 conda-forge Cached + libgfortran5 13.2.0 hf226fd6_3 conda-forge Cached + libbrotlienc 1.1.0 hd74edd7_2 conda-forge 280kB + libbrotlidec 1.1.0 hd74edd7_2 conda-forge 28kB + libxcb 1.16 hc9fafa5_1 conda-forge 325kB + glpk 5.0 h6d7a090_0 conda-forge Cached + mpfr 4.2.1 h1cfca0a_2 conda-forge Cached + libasprintf-devel 0.22.5 h8414b35_3 conda-forge 35kB + freetype 2.12.1 hadb7bae_2 conda-forge 596kB + libtiff 4.6.0 hf8409c0_4 conda-forge 239kB + libintl-devel 0.22.5 h8414b35_3 conda-forge 39kB + libgettextpo 0.22.5 h8414b35_3 conda-forge 160kB + gettext-tools 0.22.5 h8414b35_3 conda-forge 2MB + libhwloc 2.11.1 default_h7685b71_1000 conda-forge Cached + libgfortran 5.0.0 13_2_0_hd922786_3 conda-forge Cached + brotli-bin 1.1.0 hd74edd7_2 conda-forge 17kB + openjpeg 2.5.2 h9f1df11_0 conda-forge 317kB + lcms2 2.16 ha0e7c42_0 conda-forge 212kB + libgettextpo-devel 0.22.5 h8414b35_3 conda-forge 37kB + tbb 2021.12.0 h7b3277c_4 conda-forge 129kB + fftw 3.3.10 nompi_h6637ab6_110 conda-forge Cached + libopenblas 0.3.27 openmp_h517c56d_1 conda-forge Cached + brotli 1.1.0 hd74edd7_2 conda-forge 20kB + gettext 0.22.5 h8414b35_3 conda-forge 483kB + libblas 3.9.0 23_osxarm64_openblas conda-forge Cached + gawk 5.3.0 h90c4c8c_0 conda-forge 1MB + libcblas 3.9.0 23_osxarm64_openblas conda-forge Cached + liblapack 3.9.0 23_osxarm64_openblas conda-forge Cached + mafft 7.526 h99b78c6_0 conda-forge 5MB + gsl 2.7 h6e638da_0 conda-forge Cached + suitesparse 7.8.2 hec630bf_0 conda-forge 1MB + dsdp 5.8 h9397a75_1203 conda-forge Cached + tzdata 2024a h8827d51_1 conda-forge 124kB + iqtree 2.3.6 h856ad77_0 bioconda 3MB + fasttree 2.1.11 h774b16e_4 bioconda 218kB + vcftools 0.1.16 pl5321h6057758_11 bioconda 465kB + python 3.11.9 h932a869_0_cpython conda-forge 15MB + wheel 0.44.0 pyhd8ed1ab_0 conda-forge Cached + setuptools 73.0.1 pyhd8ed1ab_0 conda-forge 1MB + pip 24.2 pyh8b19718_1 conda-forge 1MB + zipp 3.20.1 pyhd8ed1ab_0 conda-forge 21kB + pycparser 2.22 pyhd8ed1ab_0 conda-forge 105kB + pyparsing 3.1.4 pyhd8ed1ab_0 conda-forge 90kB + cycler 0.12.1 pyhd8ed1ab_0 conda-forge 13kB + certifi 2024.8.30 pyhd8ed1ab_0 conda-forge 164kB + attrs 24.2.0 pyh71513ae_0 conda-forge 56kB + packaging 24.1 pyhd8ed1ab_0 conda-forge 50kB + networkx 3.3 pyhd8ed1ab_1 conda-forge 1MB + six 1.16.0 pyh6c4a22f_0 conda-forge 14kB + pytz 2024.1 pyhd8ed1ab_0 conda-forge 189kB + importlib-metadata 8.4.0 pyha770c72_0 conda-forge 28kB + isodate 0.6.1 pyhd8ed1ab_0 conda-forge 29kB + python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge 223kB + munkres 1.0.7 py_1 bioconda 10kB + pillow 10.4.0 py311hd7951ec_0 conda-forge 42MB + kiwisolver 1.4.7 py311h2c37856_0 conda-forge 59kB + pyrsistent 0.20.0 py311h460d6c5_1 conda-forge 123kB + python-zlib-ng 0.5.0 py311h2ddeff8_0 conda-forge 56kB + python-isal 1.7.0 py311hd3f4193_0 conda-forge 64kB + python-calamine 0.2.3 py311h481aa64_1 conda-forge 648kB + numpy 1.26.4 py311h7125741_0 conda-forge 7MB + cvxopt 1.3.2 py311h5e1cfd8_3 conda-forge 507kB + cffi 1.17.0 py311h3a79f62_1 conda-forge 288kB + fonttools 4.53.1 py311h460d6c5_1 conda-forge 3MB + contourpy 1.3.0 py311h2c37856_1 conda-forge 249kB + biopython 1.84 py311hd3f4193_0 conda-forge 3MB + scipy 1.14.1 py311h2929bc6_0 conda-forge 15MB + pandas 1.5.3 py311h4eec4a9_1 conda-forge 14MB + zstandard 0.23.0 py311ha60cc69_1 conda-forge 332kB + matplotlib-base 3.9.2 py311hba6b155_0 conda-forge 8MB + pyfastx 2.1.0 py311hb8cc920_3 bioconda 603kB + bx-python 0.11.0 py311hc409921_3 bioconda 1MB + jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge 46kB + xopen 2.0.2 pyh707e725_1 conda-forge 22kB + treetime 0.11.4 pyhdfd78af_0 bioconda 122kB + bcbio-gff 0.7.1 pyh7cba7a3_1 bioconda 22kB + augur 25.4.0 pyhdfd78af_0 bioconda 260kB ```

Note that mafft is pulled from conda-forge beacuse the package on Bioconda is not available for osx-arm64.

mfansler commented 2 months ago

Nice!

Just a minor note: I see the environment installs OpenBLAS by default. From what I've benchmarked, it significantly underperforms Accelerate on osx-arm64, so I would encourage to try including a 'blas=*=accelerate'.

victorlin commented 2 months ago

@mfansler thanks for sharing, that is useful info. We don't use BLAS directly in Augur – it is the numpy dependency that could potentially use it. I think the proper solution is for numpy to declare that dependency on osx-arm64. There are a few relevant issues:

but it seems that maintainers are reluctant to using Accelerate as default due to compatibility issues with SciPy.