Working Multi-Arch AMD64/ARM64 Images

[the-wondersmith]

Just posting this here in case anyone else has run into the same roadblock I did - the lack of multiarch (specifically arm64) images for emissary and telepresence.

I've put together a fully functional, docker buildx build-able Dockerfile that will builds and assembles a functional emissary image. I've been using a private GCR repo while I work this process out, but I will push the working multi-arch images (along with the same for telepresence) to my public Docker Hub within the next day or so.

DISCLAIMER: While the binaries built by this process are absolutely identical to those from "official" images, the final image produced by this build process is not tested or supported by the Datawire team. DO NOT go whining to them if it doesn't work properly or has some issue.

If you'd like to build your own, (or don't feel like waiting for me to get around to pushing the images to Docker Hub) just run:

# Prebuild Preparation Command
mkdir -p /tmp/emissary-build && cd /tmp/emissary-build && curl https://raw.githubusercontent.com/the-wondersmith/emissary/multiarch-build/Dockerfile.multiarch > /tmp/emissary-build/Dockerfile

# Single-Arch Build
docker build --tag='your.personal.repo/emissary:your-tag-here' -f /tmp/emissary-build/Dockerfile . && docker push your.personal.repo/emissary:your-tag-here

--OR--

# Multi-Arch Build
docker buildx build --push --tag='your.personal.repo/emissary:your-tag-here' --platform='linux/arm64/v8,linux/amd64' -f /tmp/emissary-build/Dockerfile .

# Deployment Command
curl https://app.getambassador.io/yaml/edge-stack/2.2.2/aes-crds.yaml > /tmp/emissary-aes-crds.yaml && sed -i 's#docker.io/datawire/aes:2.2.2#your.personal.repo/emissary:your-tag-here#g' /tmp/emissary-aes-crds.yaml && kubectl create namespace ambassador && kubectl apply -f /tmp/emissary-aes-crds.yaml && kubectl wait --timeout=90s --for=condition=available deployment emissary-apiext -n emissary-system && helm upgrade -i edge-stack datawire/edge-stack \
  --namespace ambassador \
  --create-namespace \
  --set="emissary-ingress.image.tag=your-tag-here" \
  --set="emissary-ingress.image.repository=your.personal.repo/emissary" \
  --set="emissary-ingress.agent.image.tag=your-tag-here" \
  --set="emissary-ingress.agent.image.repository=your.personal.repo/emissary" \
  --set="emissary-ingress.canary.image.tag=your-tag-here" \
  --set="emissary-ingress.canary.image.repository=your.personal.repo/emissary"

Obviously, please report any comments, questions, concerns, or feedback in this thread.

[the-wondersmith]

Working images are now up on Docker Hub for:

• emissary / edge-stack
• telepresence / traffic-manager

[richarddli]

wow, this is cool! thanks for doing this! are you running these in a production use case? any issues? (we'd love to support support this; the main part of course is running the full regression suite automatically as part of CI)

[the-wondersmith]

@richarddli I've been working with @kflynn and @LukeShu to revamp emissary's current build process to be compatible with multiarch builds (like docker buildx) and more streamlined in general. I personally am not running this in a production environment, but that's mostly a function of only haven gotten it working yesterday. That said though, there are zero changes to the actual emissary or datawire/envoy codebase(s), so at least in theory the current CI tests should cover it. There are minor differences in the final "on-disk" location of some of the binaries, but they ultimately do end up functionally identical vis-a-vis accessibility / callability.

In terms of proposed changes to the build process itself - as far as I can tell, there are really three main "stages" of the current emissary build process:

• Python related stuff
• Emissary Golang compilation
• Envoy (specifically the Datawire fork thereof) compilation

Personally my "primary" programming language is Python, and there are a few improvements I can see that would simplify how the Python stuff gets laid out in the final image. It's not huge though, and the main point would be to alleviate some of the scatteredness I've seen with where things end up on-disk.

I've discussed with Flynn the prospect of leveraging golang's built-in support for cross-compilation, and tested it locally. It's 100% possible and works without flaw or fuss on every host I've so far tried for every target I've so far tried.

That just leaves the envoy binaries that need targeted compiling, and I'm still working out a reasonable way to handle that. My first instinct is just to have the extant build system for datawire/envoy publish its own multi-arch builds using the CI/CD scripts that already exist (and are maintained by the upstream envoy team) in the repo. Assuming compatible images were available, that would simplify the envoy portion of things down to literally a single line in the Dockerfile:

COPY --from=datawire/envoy:latest /usr/local/bin/envoy-static-stripped /usr/bin/envoy

Dunno how you guys feel about any of that, but I'm obviously happy to help if I can 😅.

[the-wondersmith]

UPDATE: I've worked out Google Cloud Build configuration file for auto-magically building the datawire/envoy binaries and pushing the images to Docker Hub tagged with matching branch name the image was built from. I'm waiting on a build run to complete as I type this, and once it's done I'll be able to double-check the proposed COPY --from solution. (Spoiler: I don't anticipate it being an issue, just a massive improvement to build speed).

I'll report back shortly.

[kirinnee]

hey, @the-wondersmith , can I check what happend to the images?

[cindymullins-dw]

@the-wondersmith , would you be able to join our next Emissary Contributors session (1st Wed each month) so we could discuss this further?

[the-wondersmith]

@cindymullins-dw Apologies for the delay in responding, I'd be delighted to join the February meeting.

[the-wondersmith]

hey, @the-wondersmith , can I check what happend to the images?

@kirinnee Similar apologies for the delay in responding - I wasn't able to find time with the Emissary team to vet the images for functionality, and ended up taking them down. I'm working on getting them put back up. Hopefully the Contributors session will bear fruit 😅.

[kflynn]

@the-wondersmith Hey! We actually were looking at moving the contributor meeting to 3:30PM Eastern, rather than 1PM Eastern – you still able to make it on the 1st if it's that time? Thanks! and apologies for the confusion. 😐

[kol-ratner]

@kflynn @the-wondersmith Firstly allow me to thank you for putting together this effort. I have been following this issue and am highly interested in testing this. Has there been an update in timeline for release of the arm-based docker images to docker-hub? would like to run this on arm instances in AWS...

Thanks!

[the-wondersmith]

@cole-ratner I'm trying my darndest to make some time to tackle this again. If I'm able to get new images to build, I'll update this thread. @kflynn was working on the same, the last time I was able to connect with him. I'd bet he made it farther than I did, but odds are kinda 50/50 😅.

@kflynn Any luck on your end?

[robd003]

@the-wondersmith I've got it building! I just had to update the envoy build image.

I'm mobile right now, but can post a patch between your Dockerfile and my (slight) modifications.

[samgranieri]

I'm interested in playing with this again. @robd003 I just found your docker images. I'm going to try this out in my raspberry pi homelab

[robd003]

Here's the Dockerfile for building envoy 1.25.4: https://gist.github.com/robd003/62066753c6254b299b9b7135d254d806

When building envoy tag it like this: docker build -t local_build/envoy:1.25.4 .

Here's the Dockerfile for building emissary-ingress: https://gist.github.com/robd003/ff587c755cd360093ba7f70e3af83d98

Take note that I had to fix the housekeeping.py file so that it wouldn't crash.

[amorey]

@robd003 Thanks for providing arm64 images! That was very helpful. Do you know why your arm64 image is 2X the size of the official amd64 image?

[robd003]

@amorey I'm guessing that the amd64 image has a fully stripped emissary binary.

I can try to see what's taking up space to minimize further.

[the-wondersmith]

@amorey @robd003 Super useful tool, in case you ever run up against this sort of thing again - dive

[robd003]

@the-wondersmith very cool!

[samgranieri]

@robd003 your instructions worked like a charm. I built emissary with someone else's envoy image and it works like a charm in my raspberry pi home lab.

Friends, here's my arm64 emissary image

https://hub.docker.com/r/samgranieri/ambassador

Friends, also, please do not use my image outside of a home lab or development environment.

[gecube]

Good day!

Any chances to get ARM64 official build? I am building a distributed IoT system and I want to have a good ingress controller deployed on the edge devices... It is very sad that I can't user emissary right now.

[hibohra1398]

Hello Team, We are looking for an ARM image build for emissary ingress, currently there are multiple challenges around building the image in house as there are too many dependencies.

[aztechian]

For the main chart, at least I can put in an affinity to have the deployment not land on ARM nodes. But this is a real problem for the apiext deployment (that is buried in the CRD 😢 ). For that, I have no control of the deployment - and I find it unacceptable to hand-jam changes to the CRD every release. This means I have no way to direct the scheduler to not run apiext pods on ARM nodes. The deployment never reaches healthy, and ArgoCD will always show it as progressing or out-of-sync.

Please, prioritize one of:

1. releasing an official mult-arch build+image
2. separating the apiext deployment from the CRDs

[gecube]

@aztechian Hi! Totally agree. This sounds awful. I think the best and simplest option would be to "releasing an official mult-arch build+image"

[kflynn]

@aztechian @gecube @hibohra1398 Are any of y'all on the CNCF Slack? I have a couple of rather unofficial things I'd like to get some feedback on if anyone's willing... 🙂

And props to @the-wondersmith for his original work here!

[aztechian]

Yes on slack. let us know what channel. 👍

[kflynn]

@aztechian Just ping me directly, @ flynn.