Open allexoll opened 2 years ago
allexoll
commented
2 years ago I'm not sure what kind of test would be pertinent for this, I know travis supports aarch64 & ppc64le so it would be easy on that side.
The issue is that i'm pretty sure some other things need to be fixed in other repository for a similar test-suite as amd64 to pass for aarch64.
what kind of tests were you thinking about?
mithro
commented
2 years ago @allexoll -- I was thinking maybe seeing if you can setup builds using GitHub Actions? @umarcor might have some ideas. I know that people are using GitHub Actions to build multi-arch containers for Docker. A quick Google search turned up the following links;
@kgugala from Antmicro was also setting up self hosted runners for GitHub Actions and was looking at using the Antmicro RPi cluster stuff.
At work we have a cluster of IBM Power 9 systems running OpenShift. The problem with this environment is that nobody runs Power 9 on their desktop, and Docker Hub only offers automatic build support for the x86 architecture. This means there’s no convenient options for building Power 9 Docker images…or so I thought. It turns out that Docker provides GitHub actions that make the process of producing multi-architecture images quite simple.
GitHub
GitHub Action to configure Qemu support. Contribute to docker/setup-qemu-action development by creating an account on GitHub.
Github actions multi-architecture container guide repository. - GitHub - ahmetozer/github-actions-multi-arch-container-ghcr.io: Github actions multi-architecture container guide repository.
Run commands in a Linux container with a specific CPU architecture (armv6, armv7, aarch64, s390x, ppc64le)
Contribute to eclipse/che-jwtproxy development by creating an account on GitHub.
GitHub Community
I’m trying to migrate a project from Travis to Github Actions but I can’t find how I specify different architectures. Is this currently not possible? I would preferably run my tests on the host but I’ve also tried running in a container: test_linux_arm: runs-on: ubuntu-latest container: image: arm64v8/python:3.7-buster steps: - uses: actions/checkout@v1 - name: Test run: | python -c "import platform; print(platform.machine())" But this results in...
Medium
Building Multi-Arch Docker Images in Github Action CI
How to setup your CI/CD in a Cloud environment
umarcor
commented
2 years ago We are using dbhi/qus in hdl/containers. That allows building for arm, arm64, ppc64 and s390x. The summary is:
We can provide a variant of the conda container, for foreign architectures. That's based on Debian Bullseye. Would that be useful in the CI of this repo?
Building and deploying container images for open source electronic design automation (EDA) - containers/setup.sh at main · hdl/containers
allexoll
commented
2 years ago That seems a good idea to me. I think it would be doable with travis as is, we would just need to add a arch keyword (ref: https://docs.travis-ci.com/user/multi-cpu-architectures/).
I'm currently testing to see which packages can build as is and which cannot and what fixes would be needed, but so far it seems that many packages would need some fix to allow for aarch64 build. I'll add an issue to track the progress on that front soon, and I think the question of CI will be answered by then.
We could keep this PR as draft until then if you want?
mithro
commented
2 years ago Travis is sadly no longer usable for open source projects (and has had lots off issues) but we can use GitHub Actions using both the GitHub free provided runners and the Google provided "self hosted runners" on Google Cloud Platform;
Earthly Blog
Starting in early December, a mad dash has been underway to migrate open-source projects off of Travis CI. What happened and where should you move ...
Ars Technica
Developers furious at Travis CI's "insanely embarrassing 'security bulletin.'"
allexoll
commented
2 years ago I see. That seems like a big undertaking and i don't think i'm familiar enough with the project to be able to take it on, at least for now. I'll keep working on aarch64 support until then.
allexoll
commented
2 years ago @umarcor I believe hdlc's images are also all built for amd64, correct? See:
$ docker run --rm -it hdlc/conda "/bin/uname" "-m"
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
x86_64Building it locally also produces a x86_64 image, because I believe that the base image is also only published for x86_64.
Looking at the conda.dockerfile:
RUN curl -fsSL https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh > conda_installer.sh \
&& chmod +x conda_installer.sh \ I'm confused how that would work with qus? since the binary installer is still the x86_64 version? or am I missing some detail that would make this fit together.
But it seems that the base image is specifically set to ARG ARCHITECTURE='amd64' ...
If hdlc/container is a better place to start my effort, I'd be happy to do it over there, especially if we want to end up using their container for GA over here.
mithro
commented
2 years ago Adding support for aarch64 into these conda packages is great work! I think @antmicro and @umarcor can probably take on the making the CI up and going?
umarcor
commented
2 years ago @umarcor I believe hdlc's images are also all built for amd64, correct?
It depends on what you mean with "all". There are several images built for multiple architectures. However, all the conda|symbiflow images are currently built for amd64 only.
Building it locally also produces a x86_64 image, because I believe that the base image is also only published for x86_64.
When building it locally, there is an argument to override the architecture. However, as you found out, it needs tweaking the dockerfile/script as well.
I'm confused how that would work with
qus? since the binary installer is still the x86_64 version? or am I missing some detail that would make this fit together.
The dockerfile/script needs to me modified in order to use a different URL depending on the architecture.
But it seems that the base image is specifically set to
ARG ARCHITECTURE='amd64'...
That's an argument, which can be overriden through the --build-arg when calling docker run. Nevertheless, pyHDLC (the utilities in hdl/containers) support overriding it through arg --architecture (-a); hence, you don't need to care about it. See https://hdl.github.io/containers/dev/Utils.html#pyhdlc-build and https://github.com/hdl/containers/blob/main/utils/pyHDLC/__init__.py#L418.
If hdlc/container is a better place to start my effort, I'd be happy to do it over there, especially if we want to end up using their container for GA over here.
I created this feature branch as an starting point: https://github.com/hdl/containers/commits/conda-multiarch. So, we need to handle how to tweak the URL depending on the base/architecture (note that the names used by docker don't match the ones used by conda). Do you want to tackle it?
umarcor
commented
2 years ago As commented in https://github.com/hdl/containers/pull/54#issuecomment-1056042504, Conda containers for "foreign" architectures are available now. Hence, they can be used in CI along with dbhi/qus to build Conda for arm64, ppc64 or s390x.
However, this repository seems to be unmaintained since 2020. I'd say that the current upstream are the repos in org 'hdl'. It would be sensible to continue in hdl/conda-eda.
unixb0y
commented
1 year ago Anything new about this? F4PGA still not working in Ubuntu 22.04 VM on a M2 Pro machine.
allexoll
commented
1 year ago Fiy as a temporary way to do that, you can use Rosetta in UTM to run x86 stuff: https://docs.getutm.app/advanced/rosetta/
UTM DocumentationDocumentation for UTM virtual machines
Hi,
This PR brings what's needed in this repository to start to have some support for a potential aarch64 build for symbiflow.
This does not mean that symbiflow now supports aarch64, far from it, but since symbiflow is all of these hardwired gits, better to start somewhere.
Here we just add some switch case testing for which arch we are running on and fills the ressources name for it.
Of course, remarks welcome!
Alexis Marquet