[packaging] Add arm64 Docker images

[andrewkroh]

We enabled builds of arm64 artifacts (tar.gz, deb, rpm) when you build packages. Next we want to enable builds of arm64 Docker images by default.

Currently almost all of the packaging artifacts can be produced via cross-compiles (expect macOS installers). It would nice to do this for the arm64 Docker images too. The experimental docker buildx command would support this. I've used the command to build images and it works well.

- [ ] Update the docker builder code to support docker buildx.

- [ ] Auto-detect if buildx is supported and skip that target if it's not available.

- [ ] Make it possible to support more than arm64 (like linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6).

• [x] Update package_test.go if needed. Make sure it works on the arm64 images.
• These projects have customized packaging and probably required additional attention.
  • [x] Elastic Docker Logging Plugin
  • [x] Elastic Agent
• [x] Select proper architecture for other binaries included in images (jq, tini...)
  • [x] Heartbeat synthetics deps

- [ ] Enable buildx command on CI servers.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[andrewkroh]

One additional requirement (from the release-manager team), offer a way to not use buildx so that when you are on a native linux/arm64 you can use the regular docker build. The release-manager tool for the Elastic Stack now has an arm64 worker that Beats can use.

[v1v]

Enable buildx command on CI servers.

I already configured ansible to roll out this feature. It was merged a few days ago. I'll enable some test-infra asserts in our health check pipeline to help with the validation if the CI workers are configured as expected.

[Conky5]

One additional requirement (from the release-manager team), offer a way to not use buildx so that when you are on a native linux/arm64 you can use the regular docker build.

@mgreau asked me to clarify: this is a hard requirement for getting into the release process currently. We want to make sure the beats framework for building the arm64 images supports this so it doesn't come up as a surprise later and require additional work.

[manelpb]

any updates about this? really wanted to get the official docker image for arm

[rvincemt]

Any chance that a docker-image for elastic-agent on ARM64 can come soon? Willing to help and test. Thanks!

[jsoriano]

Added point about selecting proper architecture for binaries currently installed using curl.

[kseniia-kolpakova]

Just a reminder from infra. We need to make sure that we are not using buildx to build the ARM images, but that rather we’re on a native linux/arm64 instance (see comment : https://github.com/elastic/beats/issues/18334#issuecomment-638910015)

[kuisathaverat]

We are exploring this, and I think we are taking the wrong approach. Let's start thinking about what cross-build means, cross-build is a way to generate binaries for different architectures form commonly used architectures (usually amd64), it is easy to find tools to cross-build binaries for different architectures on amd64, but the other way around used to not be so easy or even possible. If we can not use dockerx we will build the Docker image in the same architecture we will use as target architecture, we will build the ARM Docker image in an ARM machine. On an ARM machine, you have a native Go compiler does not make sense to use a cross-compile to generate an ARM binary. Even more, on an ARM does not make sense to build any other platform because we build 'em all on an Intel host. I guess the main problem that forces us to use Docker images are the libraries, in that case, the Docker image we need is a simple golang Docker image with the libraries we need, something much simpler than the cross-compiler images we have because we do not need to cross-compile we are running in ARM architecture. If we can compile directly on the native worker I'd preferred to use native tools, so directly use go build, process the Dockerfile, and docker build in case we are in running in the sea architecture that the target binary.

Also, I'd like to ask @mgreau if using dockerx is still an issue, I know it is still experimental but it was there for a while and is extended used.

[kuisathaverat]

Also, I'd like to ask @mgreau if using dockerx is still an issue, I know it is still experimental but it was there for a while and is extended used.

It is https://github.com/elastic/infra/issues/25762#issuecomment-761111693

taking a look to https://github.com/elastic/infra/issues/25762, other teams make the packaging process in two steps first build the binaries and push them to a bucket and after that, build the Docker images in the native architecture with docker build easy peasy, the docker file manages to download the right package. This is pretty much what we do for the DoD Docker images.

[kvch]

Follow-up after we had an off-line conversion with @v1v and @kuisathaverat about how to approach this.

We decided to stay on the well-known path of building inside a Docker container for the following reasons:

• we have complete control over the whole environment
• on startup we get a clean environment
• does not interfere with other dependencies on the system
• we can reproduce previous build environments
• it is how we have been packaging all Beats, and it would require an effort to change the process for one arch

The required crossbuild image is submitted here: https://github.com/elastic/golang-crossbuild/pull/76

[kuisathaverat]

The required crossbuild image is submitted here: elastic/golang-crossbuild#76

to be clear, it is an ARM Docker image with Go, gcc, and all the libraries we need to compile beats, it compiles using the native Go, gcc, and libraries in the Docker images, but it not make any kind of cross-compile or uses cross-compile tools, it is not needed, the target architecture, the Docker host, and the Doker image have the same architecture ARM. This Docker image can only generate ARM binaries running in an ARM host.

[mgreau]

To clarify why we asked not to use Docker Buildx back in April 2020, there were 3 reasons:

1. Docker Buildx was experimental https://docs.docker.com/buildx/working-with-buildx/

   Experimental features must not be used in production environments. Docker does not offer support for experimental features.

2. Our referent at Docker Inc. (@tianon) told us they were building images on their proper architecture

   we don't do any cross-building -- we've had tons of issues with QEMU user-mode emulation historically, so we only use it for development and testing; for actual image builds they all build directly on their proper architectures. ... there's slightly more details over at https://github.com/docker-library/faq#how-are-images-built-especially-multiarch if you're curious

3. the Release Build Infrastructure in place at that time was not capable of handling this tool because of the main build happening inside Vagrant VM boxes running on bare-metal servers.

With the work done with the Release Build Improvements project over the last 5 months, (3) is no longer an issue. The Release builds happen on GCE Linux workers, macOS workers, and ARM workers.

(1) and (2) are still relevant today, and the Elasticsearch, Logstash, and Kibana teams are now all using the same workflow where the Docker images are built on ARM workers. That's why we are still advocating to not use Docker Buildx.

There are 2 different ways for handling this. Both are totally supported by the Release Build workflow:

• (a) - Build your artifacts using cross-compile feature while executing the main Beats build in phase-1 on GCE worker (Beats Jenkins job for master build). Then, during the 2nd phase happening on ARM workers, get the built artifacts from phase-1 and build the Docker images
• (b) - Build your artifacts and Docker images during the phase-2 on ARM workers

I'm not that familiar with the Beats build process. Can you clarify if your intent is to use the workflow (a) or (b) or non of the above?

Thanks

[kvch]

Currently, ARM64 Docker images can be built for all Beats on ARM64 machines with the following command:

PACKAGES=docker PLATFORMS=linux/arm64 mage package

[morganchristiansson]

arm64 docker images are available as snapshots on https://www.docker.elastic.co/r/beats?show_snapshots=true

For mutlarch image use docker pull docker.elastic.co/beats/filebeat:8.0.0-SNAPSHOT

image: "docker.elastic.co/beats/filebeat"
imageTag: "8.0.0-SNAPSHOT"

Thanks guys! :1st_place_medal:

[ph]

@kvch and @morganchristiansson Looking at the last comment I believe we can close this?

[morganchristiansson]

docker.elastic.co/beats/filebeat:7.13.0 has arm64 on https://www.docker.elastic.co/r/beats/filebeat

Can def close this :1st_place_medal:

[kvch]

We are still waiting for Elastic Docker logging plugin on ARM, so I would like to keep this open.

[kuisathaverat]

@kvch Do you mean to build the Elastic Docker logging plugin on ARM? Is it something Robots we can do? or it is something else

[kvch]

@kuisathaverat No, on our side everything is ready. Thank you. However, it is going to be a bigger undertaking from the Infra team, we are still figuring out how to proceed.

[kvch]

I am closing this issue as all arm64 Docker images are ready and released. I have created a follow-up issue for our Docker plugin: https://github.com/elastic/beats/issues/26108