search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.41k stars 5.29k forks source link

Cross Build Images for ARM #4211

Closed RichiCoder1 closed 2 years ago

RichiCoder1 commented 4 years ago

Summary

Currently ArgoCD supports building arm64 images but doesn't publish them due to issues with cross-building with normal docker and GitHub Actions (AFAIK). There's some recent advancements that would enable this with GitHub Actions.

Motivation

Running ArgoCD on arm64 via registry images.

Proposal

Long story short, there's some excellent actions that will be landing shortly: https://github.com/docker/build-push-action/pull/92#issuecomment-679094587

They, amgonst other things, make it easier to cross build using QEMU (at the price of emulation overhead): https://github.com/crazy-max/test-docker-action/blob/master/.github/workflows/full.yml#L59

Figured I'd create this ticket after doing some research and noting that the limitation mostly seemed to be docker/CI related.

ellisvalentiner commented 3 years ago

@RichiCoder1 are you looking for help with this issue?

RichiCoder1 commented 3 years ago

Hmmm. I could certainly take a swing, but the docker build is deeply baked into ArgoCD's make so I'd likely need guidance (or a long weekend which I won't have for a little bit haha)

alinbalutoiu commented 3 years ago

Another faster option would be to use a CI that supports the ARM systems. For example, Drone CI (free for open source) includes ARM servers which allows you to easily build the image. The servers are also quite powerful.

I'm currently using that one to run and check that the ARM images work successfully. On top of the release branches there is this commit that does the build for ARM images, the build history is here.

agustinvinao commented 3 years ago

Hi @alinbalutoiu, Im trying to deploy argo on a Pis cluster. I was checking the images you'd build. Are those images accessible or did you put it in a private registry?

It would be great to have ArgoCD images for Arm64 available.

thanks

ellisvalentiner commented 3 years ago

@agustinvinao I believe these are the builds

zeph commented 3 years ago

@alinbalutoiu perfectly working! thanks mate ...I'm running a microk8s cluster on RP4+ (8GB)

slmingol commented 3 years ago

Above worked w/ my k3s cluster on RPi4+ as well. Thanks for the images!

DWSR commented 3 years ago

@RichiCoder1 Any progress on using Buildx for this?

RichiCoder1 commented 3 years ago

@DWSR I haven't put any active effort into this personal, though I think some work has happened via other contributers to make this happen. I haven't (and for the forseeable future) had time to work on this

alinbalutoiu commented 3 years ago

@DWSR I'm currently trying to build with docker buildx in GitHub Actions but the build is failing when building on arm64.

It looks similar to the issue described here, trying to debug the issue to see what's wrong.

TrueBrain commented 3 years ago

Been toying a bit with this. Just leaving this here as it might help someone else.

And I think the problem @alinbalutoiu ran into: when using qemu to emulate arm/arm64, you run into the issue that yarn cannot contact the Internet (neither can npm btw). It seems not all syscalls can be emulated, resulting in networking issues. The solution seems to be to prepare a yarn cache via the native system, and share that with the build. How I approached this:

Before building, run on native system:

docker run --rm -it -v $(pwd)/ui/package.json:/src/package.json:ro -v $(pwd)/ui/yarn.lock:/src/yarn.lock:ro -v $(pwd)/cache:/cache --workdir /src node:12.18.4 yarn install --cache-folder /cache

I haven't found yet how to only download the files, so it will install everything. Shouldn't take long (17 seconds on my machine).

Now we have a cache, we can give this to the build step and build arm with this diff in Dockerfile:

+ADD cache /cache

-RUN yarn install
+RUN yarn install --offline --cache-folder /cache

The Docker context becomes huge, so not really ideal, but it does do the trick. Now building via docker buildx seems to work fine:

docker buildx build --platform linux/arm64 -t truebrain/argocd:v2.0.0-rc1 --build-arg BUILD_ALL_CLIS="false" .

This produces images; I have not tried them yet, that is next on the agenda :D

For anyone also trying this, also a few observations worth sharing:

mickkael commented 3 years ago

Been toying a bit with this. Just leaving this here as it might help someone else.

  • install-packr-linux.sh needs some patching to work with Go 1.11+. I created a PR for this: #5904.
  • install-ksonnet-linux.sh no longer works, as there is a missing repository in the ksonnet dependency. See #5271 for what to patch out to make the build work without ksonnet.

And I think the problem @alinbalutoiu ran into: when using qemu to emulate arm/arm64, you run into the issue that yarn cannot contact the Internet (neither can npm btw). It seems not all syscalls can be emulated, resulting in networking issues. The solution seems to be to prepare a yarn cache via the native system, and share that with the build. How I approached this:

Before building, run on native system:

docker run --rm -it -v $(pwd)/ui/package.json:/src/package.json:ro -v $(pwd)/ui/yarn.lock:/src/yarn.lock:ro -v $(pwd)/cache:/cache --workdir /src node:12.18.4 yarn install --cache-folder /cache

I haven't found yet how to only download the files, so it will install everything. Shouldn't take long (17 seconds on my machine).

Now we have a cache, we can give this to the build step and build arm with this diff in Dockerfile:

+ADD cache /cache

-RUN yarn install
+RUN yarn install --offline --cache-folder /cache

The Docker context becomes huge, so not really ideal, but it does do the trick. Now building via docker buildx seems to work fine:

docker buildx build --platform linux/arm64 -t truebrain/argocd:v2.0.0-rc1 --build-arg BUILD_ALL_CLIS="false" .

This produces images; I have not tried them yet, that is next on the agenda :D

For anyone also trying this, also a few observations worth sharing:

  • Building for arm/v7 fails for me on go get with TLS errors for any Google domain (works fine for github.com if using GOPROXY=direct). The reason seems to be that emulation is so slow, go get gives up before the TCP connection is established. It also takes ~60 CPU minutes to get to that point (where arm64 only takes a few CPU seconds to get there). This can fully be a problem of my Qemu version etc (using Ubuntu 20.04 via WSL2).
  • The yarn steps take a while on arm64. This seems to be because it needs to compile a lot of these dependencies, as many don't seem to have an arm64 precompiled variant.
  • webpack on native machines takes 133 seconds (enable timestamps). Via emulation, it takes even longer (800+ seconds for me). Be patient. It will get there in the end.

instead of building the cache externally, I've used the following arg with yarn --network-timeout 100000 it works for me on a rpi4 host.

TrueBrain commented 3 years ago

instead of building the cache externally, I've used the following arg with yarn --network-timeout 100000 it works for me on a rpi4 host.

Nice, that works fine too indeed :) Much easier solution ;)

Small status update:

Other than that, this seems to work fine on a raspberry pi 4 cluster :)

agustinvinao commented 2 years ago

Hi,

I was trying to find any info about when arm64 images may be available. Any update on this or an idea when this may be available?

Im5tu commented 2 years ago

I'm also keenly interested in this. Anything that I can do to help push this along? :)

johnlonganecker commented 2 years ago

My Solution - Build with docker on Pi4 8G running Raspian

I realize this doesn't answer the issue, but when I google for arm images I get this page so I am including these directions to help future me or someone else on how to build and use an arm image

Install docker

curl -sSL get.docker.com | sh && \
  sudo usermod pi -aG docker

Built the image (this took awhile)

git clone https://github.com/argoproj/argo-cd.git
cd argo-cd
git checkout v2.1.2
make armimage
docker tag argocd:stable-arm johnlonganecker/argocd:v2.1.2
docker push johnlonganecker/argocd:v2.1.2

Installed with getting started yaml

wget https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Replaced container images with arm images in install.yaml

image: ghcr.io/dexidp/dex:v2.27.0      -> image: dexidp/dex:v2.27.0
image: quay.io/argoproj/argocd:v2.1.2  -> image: johnlonganecker/argocd:v2.1.2
kubectl apply -n argocd -f <modified-install-yaml>

Then it just worked!

alborotogarcia commented 2 years ago

image: ghcr.io/dexidp/dex:v2.27.0 -> image: dexidp/dex:v2.27.0

@johnlonganecker Looking at the github dex repo it seems that there are arm64 builds at v2.27.0.. what makes them different from the ones at docker registry?

I got it working on my pi4, though I am not sure if these are built on top of qemu, if so, then there's a big trade off in performance for sure..

johnlonganecker commented 2 years ago

@alborotogarcia When I left the image as ghcr.io/dexidp/dex:v2.27.0 it was not pulling an arm image and would fail to get into a running state. When I pulled it from docker hub IE dexidp/dex:v2.27.0 it just worked no problem. This is referencing the dexidp dockerhub account, I am not sure how they built their images (I am assuming that is what you meant by your last comment?).

I did not try throwing on the @ sha256 to get it to work, but that is worth trying! https://github.com/dexidp/dex/pkgs/container/dex/529012?tag=v2.27.0

rdelpret commented 2 years ago

Built a v2.1.3 Image yesterday on my 4 GB Raspberry pi and put it up on docker hub: rdelprete/argocd-arm64:v2.1.3

It took a very long time, I had to add --network-timeout 1000000 to all the yarn commands 😮

robinhuiser commented 2 years ago

I had to update the yarn install command with:

RUN for t in 1 2 3 4 5; do yarn install --network-timeout 1000000 && break || sleep 1; done

...since it would just not finish the first time (I am building argocd using Concourse pipelines deployed on a Microk8s Raspberry pi cluster).

Problem was... first time.. you do not have anything cached - hence (as many people are hitting the same issue) - you need to re-run "manually" the yarn install command a couple of times. Which is hard in a pipeline.