search

antrea-io / antrea

Kubernetes networking based on Open vSwitch
https://antrea.io
Apache License 2.0
1.65k stars 362 forks source link

Containers for ARM architecture #450

Closed tomk235 closed 3 years ago

tomk235 commented 4 years ago

Describe the problem/challenge you have Hi. as the containers that the installer adds are x86 they won't run on a raspberry. Would just be nice to have them available for ARM.

Describe the solution you'd like Possibility to choose ARM architecture or instructions to create the containers for ARM architecture from source.

Thanks, Tom

antoninbas commented 4 years ago

@McCodeman what should be the priority for this item in your opinion?

McCodeman commented 4 years ago

@antoninbas how much time do we estimate repackaging for ARM would take? Would any special tuning be required for rasberry pi platform. Is there an IOT, edge, or other use case for this? @tomk235 can you provide us some additional user story context so we can prioritize appropriately.

@tomk235 we could also work with you to update our Makefile and build process to produce the cross compiled binaries and appropriate arm docker packaging. Is this something you would like to take on?

McCodeman commented 4 years ago

@tomk235 which arm architectures are you targeting: ARMv4, ARMv5, ARMv6, ARMv7, ARMv8? We need to see which architectures support cross compile for OVS components as well.

antoninbas commented 4 years ago

@McCodeman It seems fairly easy to build Docker images for different architectures in practice (https://docs.docker.com/docker-for-mac/multi-arch/). But it's hard to give a time estimate. If everything goes smoothly it will just take a couple hours to set everything up. But if we run into issues, e.g. when building OVS, it could take much longer to figure things out.

antoninbas commented 4 years ago

I'm trying this in the background. Will report back here if there is any issue.

Note that we currently use Ubuntu as the base image and that only supports ARMv6 onwards:

linux/386
linux/amd64
linux/arm/v7
linux/arm64/v8
linux/ppc64le
linux/s390x

This is why Raspberry Pi 1 (ARMv6) cannot run Ubuntu, but RPI 2(ARMv7) can. On my side I am reluctant to change the base image just to support RPI 1...

antoninbas commented 4 years ago

After 1+ hour running on my laptop, the base image for Antrea (which includes OVS) is not even close to being built (it hasn't even started building OVS itself yet). It seems that the emulation is making this painfully slow. This tells me that if we use docker buildx:

antoninbas commented 4 years ago

Currently investigating using Travis CI for this, as it offers free arm64 builders: https://docs.travis-ci.com/user/multi-cpu-architectures/

antoninbas commented 4 years ago

https://github.com/vmware-tanzu/antrea/pull/520 adds support to build and push Antrea docker images for ARM architectures for every Antrea release. Unfortunately, I couldn't find a simple way to test these images using public CI services. Travis CI does provide arm64 workers (https://docs.travis-ci.com/user/multi-cpu-architectures/), but the workers are LXD containers and they cannot run Kind (https://docs.travis-ci.com/user/multi-cpu-architectures/#security-and-lxd-container).

antoninbas commented 4 years ago

I suggest we discuss this at the next Antrea community meeting (03/25).

antoninbas commented 4 years ago

This was discussed at the Antrea community meeting on 03/25/2020 - see meeting minutes.

vrabbi commented 3 years ago

Any updates on this?

antoninbas commented 3 years ago

I haven't spent any time on this in a while. Still unsure what's out best solution to have a CI pipeline to build & test the ARM Docker images. We are definitely looking for help on this one, and we would love to collaborate with projects like k3s and microk8s to validate Antrea as a CNI for these distributions and integrate with their CI pipelines for testing.

rhenwood-arm commented 3 years ago

Hi @antoninbas : there are some hardware platforms that may be suitable for your needs. Can you provide more detail on what you're looking for?

antoninbas commented 3 years ago

Hi @rhenwood-arm. We are looking for a solution to address the following:

vrabbi commented 3 years ago

Hey @antoninbas, have you looked at the new AWS arm based servers - graviton? They are much cheaper than the x86 servers and are really great performance. The only thing is I don't know of a arm v7 solution on AWS but it is a great arm v8 solution

antoninbas commented 3 years ago

@vrabbi Yes I'm aware of the armv8 servers on AWS. I'm still hoping for a solution that supports multiple arm versions. It seems that kops does support using these servers as worker Nodes: https://github.com/kubernetes/kops/pull/8938, so that would be a good solution for CI. I would still like to hear from people about whether they only care about armv8 or whether they also care about other ARM flavors.

rhenwood-arm commented 3 years ago

@antoninbas: There is an Armv7 and Armv8 option that I'm looking into. I hope to have some details in the next couple of days.

ppouliot commented 3 years ago

Everyone,

There are free Arm VM resources available for OSS projects located here: https://osuosl.org/services/aarch64/

The request form for access is located here: https://osuosl.org/services/aarch64/request_hosting/

If there are questions feel free to contact me directly.

rhenwood-arm commented 3 years ago

@ppouliot : do the platforms include Armv7 hardware support?

rhenwood-arm commented 3 years ago

I've chatted to @ppouliot - and I believe these platforms do include Armv7 (aka AArch32) hardware support - so they could be the perfect fit.

@antoninbas are you blocked in other places with your Arm enablement?

antoninbas commented 3 years ago

@ppouliot @rhenwood-arm thanks for sharing this. I don't see any mention of Armv7 / AArch32 though. Am I missing something?

Edit: or do you mean that the AArch64 servers they offer can also run AArch32 binaries natively?

rhenwood-arm commented 3 years ago

@antoninbas : yes, in discussion with @ppouliot he has advised that the Arm hardware behind the OSUOSL cluster includes hardware support for AArch32. I thought this would be easy to verify by looking up spec sheets etc - but after 30 minutes of clicking around and reading specifications, the best I got was: OSUOSL cluster hardware Ampere eMag. Ampere eMag is an Armv8.0-A platform. Armv8.0 specification[1] adds AArch64 as an extension to AArch32 (Armv7) - so typically, Armv8.0 hardware has AArch32 support. Further reading suggests that this isn't always the case [2]... so I stopped there and started writing this response.

Sorry I couldn't be more specific.

  1. https://web.archive.org/web/20190101024118/https://www.arm.com/about/newsroom/arm-discloses-technical-details-of-the-next-version-of-the-arm-architecture.php
  2. See Cortex-A32 and Cortex-A34 here: https://en.wikipedia.org/wiki/Comparison_of_ARMv8-A_cores
antoninbas commented 3 years ago

I'm happy to announce that Project Antrea has received some virtual machines hosted on ARM64 hardware, which we will use to build Antrea docker images for ARM and run some basic tests on them. Thanks to @ppouliot for supporting our request, to Ampere for providing the hardware and to OSU Open-Source Lab for hosting the VMs.

antoninbas commented 3 years ago

The antrea/antrea-ubuntu Docker image is now a multi-arch manifest that supports amd64, arm64, and arm/v7. The only remaining item before we can close this issue is to update the documentation in time for the 0.14 / 1.0 release.

antoninbas commented 3 years ago

Closing this issue. The Antrea v1.0 release is imminent. The documentation has already been updated in "main".

The process of deploying Antrea to a K8s cluster is the same, whether it uses x86 Nodes, ARM (arm64 or arm/v7) Nodes, or a mix of both.