Suggestion: Consider making Aarch64 builts ?

DrYak commented 4 years ago

A weird outlandish suggestion. Given that:

Apple is building future laptops with their own homemade ARM CPUs (Mac Books - the most popular laptop among devs that aren't hardcore followers of the way of the Penguin :-D )
A Fujistu's ARM 64-based super computer is in the top 500.
Raspberry foundation has released a Pi board that has 8GB RAM (This begins to be quite descent to run some simple bioinformatics tasks, while being easy to be used by students, thanks to be being are cheap and ubiquitous)
and other SBC manufacturer are even building clusters of such boards (making a possible environment to build and test Aarch64 code).
Upstream Conda-Forge already has Linux Aarch64 builds.

...would the bioconda project eventually consider enabling Aarch64 builds of its package ?

epruesse commented 4 years ago

Eventually? Yes. Today? No.

You are welcome to get the necessary infrastructure in place to support that kind of build, we'd welcome a PR. Many packages require extra effort for every build target though, and unless there is a real need, it's difficult to justify. Just MacOS in addition to Linux already is a burden.

mr-c commented 4 years ago

@epruesse Thanks for the response. What's the process to add arm64 / aarch64 support to an existing recipe? We could document that, for users who want to self-service.

epruesse commented 4 years ago

We can't just add it for a single recipe. You'd have to start by creating the infrastructure parts for the new build target. Then you'd try to build all 6000 recipes. Many will fail, so you will have to fix them, making sure that they still build under both other target platforms. Once you have a reasonable set of packages done, say 5000 or so, you could give up and mark the rest with skip: True # [aarch64]. But you will be touching thousands of packages to get this on the road.

JeffUnderhill commented 4 years ago

@DrYak Your original post also missed that Arm-based servers are also now readily available in the AWS cloud with Graviton2. I can help with access to native Arm64 build infrastructure if that will help someone work on building/testing of Arm64? As a first pass it would be useful to get a view of how many recipes may just work and then analyze the characteristics of those that fail to see if there are any patterns to the failure modes that would help address 'blocks' of failures.

epruesse commented 4 years ago

For official packages, we use free public services only. Initially it was TravisCI, now CircleCI and Github Actions for the bulk bioconductor packages. Conda-forge is using Azure. I assume that they are using a cross compiler. Personally I think we have a lot of other areas that need work, but if you guys want to work on this, as I said, you are welcome. Head over to bioconda-utils, make a fork, set up some build chain. If in doubt you can always post the packages to a custom conda channel to start with.

bgruening commented 4 years ago

As a first pass it would be useful to get a view of how many recipes may just work and then analyze the characteristics of those that fail to see if there are any patterns to the failure modes that would help address 'blocks' of failures.

That is true. @JeffUnderhill you could for example try to rebuild bioconda on those AWS instances and see how far you will come.

mjsteinbaugh commented 3 years ago

Circling back to this, I'm up for volunteering some of my time to work on aarch64 (ARM) builds.

tfenne commented 2 years ago

Seems like this is also under discussion in https://github.com/bioconda/bioconda-utils/issues/706 with more recent comments.

jeffreykstone commented 2 years ago

Yes please!

corneliusroemer commented 2 years ago

Notably, conda-forge has an osx64-arm channel - so it would be great if this could get off the ground for bioconda, too.

At some point in the near future, almost all users of osx will be using apple silicon - at which point it would be weird to use emulation for all of bioconda.

Are there free-to-use resources now to build for osx-ARM? Things may have changed a lot since the first comments ~2y ago.

colinbrislawn commented 2 years ago

Are there free-to-use resources now to build for osx-ARM?

osx-arm64 support is on the roadmap for GitHub actions! :octocat: More discussion here.

corneliusroemer commented 1 year ago

Per Github roadmap, osx-arm64 support is now scheduled to arrive in Q3 of 2023 🎉

However, this wouldn't help with Linux aarch64. Is there a reason that we need to use Github actions? Isn't the current CI Azure based anyways?

Yikun commented 1 year ago

Just curious, how many build resources are needed to support bioconda linux aarch64?

Before the Github-hosted host supported linux aarch64, the self hosted runner or run-on-arch-action (qemu based) Could it be an alternative as aarch64 build resource?

martin-g commented 1 year ago

I use the following script to build Bioconda packages on Linux ARM64:

#!/usr/bin/env bash

# a Conda virtual environment to keep the things clean and tidy
conda activate biotest

YEAR=$(date +%Y)
MONTH=$(date +%m)
DAY=$(date +%d)
HOUR=$(date +%H)
MINUTE=$(date +%M)

LOG_FOLDER=logs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}
mkdir -p ${LOG_FOLDER}

BIOCONDA_RECIPES_ROOT=${1:-$PWD/recipes}
DOCKER_BASE_IMAGE="ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64"
MULLED_CONDA_IMAGE="ghcr.io/martin-g/create-env-aarch64"

for recipe in $(ls ${BIOCONDA_RECIPES_ROOT}); do
    echo -e "\n\nGoing to build recipe '$recipe'";

    bioconda-utils build \
        --docker \
        --mulled-test \
        --docker-base-image ${DOCKER_BASE_IMAGE} \
        --mulled-conda-image ${MULLED_CONDA_IMAGE} \
        --packages ${recipe} \
            2>&1 | tee ${LOG_FOLDER}/${recipe}.log
done

So far I have built 100+ recipes. Some of them fail, some pass, but the tooling (bioconda-utils) seems to work just fine!

phiweger commented 11 months ago

Entering Q4 of 2023 -- any update on this issue?

martin-g commented 11 months ago

Entering Q4 of 2023 -- any update on this issue?

phiweger commented 11 months ago

Entering Q4 of 2023 -- any update on this issue?

feat: add initial support for arm builds #40550

Make base images multiarch bioconda-containers#55 and Add aarch64 base images, update to Debian 12.2 bioconda-containers#58

so nearly there, awesome.

pditommaso commented 11 months ago

Can't wait for this!

martin-g commented 11 months ago

Here is the updated version of the script I shared earlier. Now it uses the official Docker images:

#!/usr/bin/env bash

conda activate biotest

YEAR=$(date +%Y)
MONTH=$(date +%m)
DAY=$(date +%d)
HOUR=$(date +%H)
MINUTE=$(date +%M)

LOG_FOLDER=logs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}
mkdir -p ${LOG_FOLDER}

BIOCONDA_RECIPES_ROOT=${1:-$PWD/recipes}
RECIPES=$2
if [ -z $RECIPES ]; then
        RECIPES=$(ls ${BIOCONDA_RECIPES_ROOT})
fi
echo "Recipes: $RECIPES"

# New official Docker images that support Linux ARM64 !
DOCKER_BASE_IMAGE="quay.io/bioconda/bioconda-utils-build-env-cos7-aarch64"
MULLED_CONDA_IMAGE="quay.io/bioconda/create-env"

for recipe in $RECIPES; do

                echo -e "\n\nGoing to build recipe '$recipe'";

                bioconda-utils build \
                        --docker \
                        --mulled-test \
                        --force \
                        --docker-base-image ${DOCKER_BASE_IMAGE} \
                        --mulled-conda-image ${MULLED_CONDA_IMAGE} \
                        --packages ${recipe} \
                        2>&1 | tee ${LOG_FOLDER}/${recipe}.log
done

corneliusroemer commented 10 months ago

There's a lot of progress! Looks like aarch64 will happen in the next few weeks. And eventually also arm64. See:

https://github.com/bioconda/bioconda-docs/pull/16

pditommaso commented 10 months ago

Can't wait for it to happen!

rob-p commented 9 months ago

Is this supported yet?

emiliofernandes commented 9 months ago

I see there is a lot of progress in the last weeks! But it is not very clear to me whether there are any stoppers at the moment. I'd be glad to help with testing and/or coding if such help is needed !

Adamtaranto commented 7 months ago

@martin-g Does this go in the build.sh script in a Bioconda recipe?

Here is the updated version of the script I shared earlier. Now it uses the official Docker images: ...

martin-g commented 7 months ago

@Adamtaranto No! This is a custom shell script that I use locally to build/test recipes on Linux ARM64. Once Bioconda officially adds the support for Linux ARM64 then you will just need to add to following to a recipe:

extra:
  additional-platforms:
    - linux-aarch64

See https://github.com/bioconda/bioconda-utils/pull/923

corneliusroemer commented 6 months ago

Per the docs, linux aarch64 now appears to be supported.

https://github.com/bioconda/bioconda-docs/pull/16/files

erikalmecdc commented 4 months ago

There is no support for linux-aarch64 for sra-tools, a vital package for any bioinformatician. Will these be added in the near future?

martin-g commented 4 months ago

@erikalmecdc sra-tools depends on many perl-** packages which need to provide linux-aarch64 builds ... https://github.com/bioconda/bioconda-recipes/pull/46237

jmarshall commented 2 months ago

As per https://github.com/bioconda/bioconda-recipes/issues/33333#issuecomment-2221176852, the infrastructure is now in place for aarch64 builds for both Linux and macOS.

bioconda / bioconda-recipes

Suggestion: Consider making Aarch64 builts ? #23454