ARM Support

[krvarma]

I am trying to install KEDA on K3S running on RPi 4, it seems the latest version does not support ARM. The kedacore/keda:arm seems to be dead and is not available from Docker Hub. I also tried to compile it manually but I ran into many issues compiling Operator-SDK.

ARM support is necessary nowadays since most of the Edge devices are ARM-based. It would be great if the ARM image can be made available on Docker Hub.

Open items

• [x] Automatically build container image on ARM for PRs
• [x] Automatically run unit tests on ARM
• [x] https://github.com/kedacore/keda/issues/2262
• [x] https://github.com/kedacore/keda/issues/2263

[tomkerkhove]

@zroubalik Are you aware if Operator SDK supports ARM? Just checking to know if this is because of issues on our side or an unsupported scenario.

[tomkerkhove]

@krvarma That's certainly a valid ask, I'm not sure why support has been removed or if it was on purpose.

Do you know @ahmelsayed @jeffhollan ?

[krvarma]

@tomkerkhove Finally I managed to run on Raspberry Pi after several failed attempts. Here is what I have done to make it work on RPi. I am not sure this is the way we have to do but I got it working this way. I compiled the KEDA on macOS instead of directly on Raspberry Pi since compiling Operator-SDK on RPi generates lots of errors. After compiling I copied the build folder to RPi.

Here are the steps

1. Set env ARCH=arm
2. make build
3. Copy the build folder to the Raspberry Pi
4. The Dockerfile in the build folder uses registry.access.redhat.com/ubi8/ubi-minimal as the base image. But this image is not available for ARM. So I modified the Dokerfile and changed to alpine:latest as the base image.
5. I build this image on Raspberry Pi and published to docker.io/krvarma/keda:latestarm (krvarma is my username on Docker Hub)
6. Set env VERSION=latestarm
7. Then I installed on my Raspberry Pi Cluster (created using K3S) using helm. (helm install . --set image.keda=krvarma/keda:$VERSION,image.metricsAdapter=krvarma/keda-metrics-adapter:$VERSION,image.pullPolicy=IfNotPresent)

My aim is to install KEDA on a hybrid cluster (AMD64 and ARM), I am trying to create docker manifest and publish to docker so that it will be downloaded based on platforms.

Let me know your thoughts

[zroubalik]

@krvarma that's great. Yeah cross-compilation is definitely recommended way. Have you tried tools like docker buildx or similar to build the arm image on your machine locally? Once it is functional, we can include it to our CI system a produce arm images during a release.

[krvarma]

@zroubalik Thanks!, I didn't try docker buildx, I will try and get back to you.

[mohsinonxrm]

@krvarma , can you give an update how is KEDA working on your Raspberry Pi cluster? I also have k3s and am planning to deploy your KEDA docker image on it.

[krvarma]

@mohsinonxrm As I mentioned in the previous comment, I compiled the KEDA source on Mac and built the image for ARM and pushed it to my Docker Hub.

My RPi cluster has 1 Master (RPi 4 4GB) and 2 nodes (RPi3 B+). The KEDA is working nicely on my cluster for about 2 months continuously without any issue. It is scaling an Azure Function that gets triggered by a RabbitMQ trigger.

I didn't face any issue with the setup.

[krvarma]

@tomkerkhove @zroubalik The image registry.access.redhat.com/ubi8/ubi-minimal:latest does not support ARM architecture. The KEDA seems to be working correctly on alpine. Is there any particular reason for using the registry.access.redhat.com/ubi8/ubi-minimal:latest?.

By using alpine you can support all the architectures.

[mohsinonxrm]

@krvarma , I'm assuming you've used helm v2 instead of v3? how did you get tiller to deploy to the cluster?

Nevermind, found ARM image for tiller.

For others who might run into the same problem, you don't necessarily need helm to deploy KEDA. You can just use the yaml files to deploy using kubectl.

However, on Raspberry Pi, you'll need krvarma/keda:localarm and krvarma/keda-metrics-adapter:localarm images

[krvarma]

@mohsinonxrm I am using Helm 3, not Helm 2.

Great you got it working on your Cluster 👍

[mohsinonxrm]

@krvarma , yes it is working great. Thanks for compiling the ARM version.

Now I'm going to deploy the durable scalar and test it.

[natdan]

+1 for Arm and even more for Arm64 support! I've found harakka/keda docker images on docker hub that are supporting keda and keda-metrics-apiserver images which deploy without issues on Arm64, but then struggled with Python3 support (azure-functions/python:3.0-python3.7 image) for Arm64. There is one and when I try to build it myself it fails at some point...

So, again can we have Arm64 support, please? Even microk8s support keda as 'extensions' but not on Arm64 :(

[zroubalik]

@natdan for azure-functions ARM support you should raise an issue in the particular azure fn repository, there's nothing we can do from KEDA here.

[natdan]

@zroubalik I know exactly what you're saying for azure-functions but there are two other ways to think about it:

• one can deploy non azure-function with KEDA anyway and benefit of scaling, etc...
• one can use different (if not homebrew/manually compiled) image for azure-functions and benefit from KEDA

There's already image in docker hub under harakka/keda (https://hub.docker.com/r/kedacore/keda) - it would be really nice if KEDA becomes Arm64 ready ahead of big changes in well known cloud computing world :)

[zroubalik]

@natdan absolutely agree on ARM support for KEDA, I was referring to your problems with azure-functions/python:3.0-python3.7 image :)

Are you willing to contribute the ARM build? My only concern is the actual testing of ARM images, we will need ARM based runner for GH Actions.

[natdan]

@zroubalik fair enough. I can't promise (that I can find another lazy Sunday like yesterday very soon) - but I'm happy to give it a go. As for Arm node - I've got couple of Raspberry Pi 4's with 64 OS on it to try it on. If someone else is to do it - we can try to make an arrangement of a sort, too.

[zroubalik]

We will need a pernament env, that we can use to run the tests periodically. Maybe CNCF could help with this? @tomkerkhove

[tomkerkhove]

Digging into this as part of https://github.com/kedacore/governance/issues/12 to provide a self-hosted runner.

[basilfx]

In the mean time: I've been using Docker Buildx to cross-compile ARM64 images on a x64 machine using Qemu and binfmt_misc (instructions). For the ones interested, I've applied the following changes to the source, then just ran IMAGE_REPO=https://some-arm64-k3s-cluster make publish and ended up with ARM64 images in my registry. Much faster and easier than compiling on a Raspberry Pi itself.

[tomkerkhove]

Just as an update - We are getting an ARM machine to run everything on. Stay tuned.

[mohsinonxrm]

built the arm64 images here: https://hub.docker.com/repository/docker/mohsinonxrm/keda https://hub.docker.com/repository/docker/mohsinonxrm/keda-metrics-apiserver

but looking forward to the above-mentioned solution.

[nlamirault]

Any news on this feature ?

[zroubalik]

@jeffhollan ^

[mohsinonxrm]

Latest linux/arm64 2.4.0 images here: https://hub.docker.com/repository/docker/mohsinonxrm/keda https://hub.docker.com/repository/docker/mohsinonxrm/keda-metrics-apiserver

@tomkerkhove , when is that ARM machine coming online? :)

[tomkerkhove]

When we have some more info from @jeffhollan on how to set it up.

[febus982]

Hi @mohsinonxrm Thanks for building the images! I have tried your 2.4.0 images but I am getting this error on the keda-operator pod:

flag provided but not defined: -enable-leader-election
Usage of /keda:
  -health-probe-bind-address string
        The address the probe endpoint binds to. (default ":8081")
  -kubeconfig string
        Paths to a kubeconfig. Only required if out-of-cluster.
  -leader-elect
        Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
  -metrics-bind-address string
        The address the metric endpoint binds to. (default ":8080")
  -zap-devel
        Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
  -zap-encoder value
        Zap log encoding (one of 'json' or 'console')
  -zap-log-level value
        Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
  -zap-stacktrace-level value
        Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').

I am deploying it with the helm chart (which works fine on minikube using 2.4.0 version).

[mohsinonxrm]

@febus982 , I think since I compiled from the main branch, it may have included some changes after 2.4.0 release. I'll recompile with 2.4.0 release source code and go from there.

BTW, this is totally unofficial and not supported.

[tomkerkhove]

Self-hosted ARM64 runner for GitHub Actions is available.

runs-on: ARM64

Anyone up for a contribution on changing our CI & release workflows?

[tomkerkhove]

Relates to https://github.com/kedacore/governance/issues/12

[febus982]

@febus982 , I think since I compiled from the main branch, it may have included some changes after 2.4.0 release. I'll recompile with 2.4.0 release source code and go from there.

BTW, this is totally unofficial and not supported.

@mohsinonxrm Thanks for the quick reply. I wasn't sure if I was missing anything 😅

[zroubalik]

@mohsinonxrm are you willing to implement this for our oficial releases? We have ARM runner, see: https://github.com/kedacore/governance/issues/12

[tomkerkhove]

Unfortunately we'll need to install some dependencies on the VM such as Docker

[mohsinonxrm]

@zroubalik , I'd like to but I don't have too much expertise with GitHub Actions. I'll give it a try though.

[mohsinonxrm]

@febus982 , I think since I compiled from the main branch, it may have included some changes after 2.4.0 release. I'll recompile with 2.4.0 release source code and go from there. BTW, this is totally unofficial and not supported.

@mohsinonxrm Thanks for the quick reply. I wasn't sure if I was missing anything 😅

Rebuilt the images using v2.4.0 tag

[febus982]

@febus982 , I think since I compiled from the main branch, it may have included some changes after 2.4.0 release. I'll recompile with 2.4.0 release source code and go from there. BTW, this is totally unofficial and not supported.

@mohsinonxrm Thanks for the quick reply. I wasn't sure if I was missing anything 😅

Rebuilt the images using v2.4.0 tag

Hero! It's working perfectly now 🙌

[zroubalik]

@mohsinonxrm awesome!

[mohsinonxrm]

while we wait for 2.6.0 release, here are 2.5.0 arm64 versions: https://hub.docker.com/repository/docker/mohsinonxrm/keda https://hub.docker.com/repository/docker/mohsinonxrm/keda-metrics-apiserver

[tomkerkhove]

Like @zroubalik said, we are open for contributions to help close this upstream in KEDA itself :)

[mohsinonxrm]

@tomkerkhove , I had a look at GitHub Actions and I'll be honest, I don't think I'm familiar enough with it to modify to use build for arm64.

BTW, the only change I made for building these images is to update the Docker and Docker.adapter files with arm64 golang images and then in the makefile change the arm to ARCH ?= arm64.

I've also installed all the build tools locally on the PI 4 or Ampere VMs (Oracle), so I'm sure the build-tools.Dockerfile can also be updated. OperatorSDK is available for arm64 as well, build Azure CLI locally... everything works. Just don't know how to put it together in a parameterized way for GitHub Actions.

Wish I could help more and take it to the finish line.

[mohsinonxrm]

@tomkerkhove , actually... let me take another look at it. I was approaching this the wrong way.

[zroubalik]

It is not about the build tools I'd say. But more about building KEDA for arm64, start a small Kind (or similar k8s distro) on ARM machine and run some smoke check (a couple of e2e tests, eg. cron or additional simple ones).

[mohsinonxrm]

I have a k3s cluster, so I'll play on that first, will report my findings here.

[tomkerkhove]

Thank you @mohsinonxrm - I will just assign this to me to keep track of this but feel free to keep on tinkering around!

[morganchristiansson]

One way to simply solve this is to cross-compile the binaries first, then use docker buildx build --platform linux/arm64,linux/amd64 and COPY the $TARGETARCH-specific binaries (instead of compiling it in Dockerfile).

I saw another go project using https://github.com/mitchellh/gox to cross-compile binaries.. It will then be simple and fast to create the multi-arch image with the buildx build command (using qemu).

Another way is to build amd64 and arm64 docker image in separate jobs and then use docker manifest to create the multi-arch manifest.

[morganchristiansson]

Very nice! Is new release being made?

[tomkerkhove]

It will be in 2.7

[JorTurFer]

taking into account our release plan, in 2 months approx will be the next release

[JorTurFer]

You can try it using main tag but we not recommend that on prod environments because main is not stable version

[DevOpsBoondoggles]

@JorTurFer Is it possible to leave this open until it's actually released maybe ? It would be helpful for people searching the same problem. I nearly missed it because it's closed but it's not actually solved in terms of me being able to use it. Thanks

[tomkerkhove]

We have closed it because the work is done but you can see that it's in "ready to ship" status of our backlog.

Otherwise it is hard to track what is open and what not.