Add images for ARM 64

[atlipall]

The new Apple M1 machines are rolling out. Fast, silent and cool. Your images do not work on them, a build for ARM 64 would be very use full.

[chetanyachopra]

broker         | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100003883970000
broker         | ===> Launching ...
broker         | ===> Launching kafka ...
broker         | qemu: uncaught target signal 11 (Segmentation fault) - core dumped

Please find the logs... broker image is not compatible with ARM

[teddyknox]

Please!

[turkmenkaan]

This would help a lot!

[alexcu]

Note that confluentinc/cp-zookeeper:6.1.1 no longer causes a segmentation fault, however the EventThread shut down still occurs:

$ docker network create --driver bridge my_network
$ docker run --platform="linux/amd64" -p "2181:2181" --name="zookeeper" --env-file=zookeeper.env --network=my_network --detach confluentinc/cp-zookeeper:6.1.1
$ docker run  --platform="linux/amd64" -p "29092:29092" -p "9092:9092" -p "9101:9101" --name="broker" --env-file=broker.env --network=my_network confluentinc/cp-kafka:6.1.1
[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.22.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.22.0.3:58248, server: zookeeper/172.22.0.2:2181
[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.22.0.2:2181, sessionid = 0x100000103610000, negotiated timeout = 40000
[main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x100000103610000 closed
[main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100000103610000
===> Launching ... 
===> Launching kafka ... 
[2021-06-07 01:10:36,273] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-06-07 01:10:37,355] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-06-07 01:10:37,658] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-06-07 01:10:37,676] INFO starting (kafka.server.KafkaServer)

Files:

• broker.env
• broker.log
• zookeeper.env
• zookeeper.log

[andrewegel]

There is currently no Confluent Platform ARM support - So solutions will have to be through the community until that support is rolled out.

In leu of duplicating the same ask, lets re-direct +1s to https://github.com/confluentinc/common-docker/issues/117 as a feature request for ARM support (Since things should start at the base images).

[hurtadosanti]

@mispon, @andrewegel can you clarify what is the solution? My understanding is that there are not arm64 images yet, as mention here https://github.com/confluentinc/common-docker/issues/117 and there are no images on docker hub but I am wondering why this issue is closed, thanks.

[mispon]

@hurtadosanti, as I understood, adding arm support is a hard task and in hold status now, so you can look for other kafka images on the docker hub that support it

[andrewegel]

It doesn't make sense to have multiple open issues for the same thing - If you follow the issue chain, you'll arrive at https://github.com/confluentinc/common-docker/issues/117 which is the base image for all cp-docker images which is open. that is the issue to track for ARM64 images with CP.

[gerbermichi]

I found a solution to build the Kafka image for other CPU architectures, such as ARM64 CPUs. https://nxt.engineering/blog/kafka-docker-image

[murphyj]

I found a solution to build the Kafka image for other CPU architectures, such as ARM64 CPUs. https://nxt.engineering/blog/kafka-docker-image

Thanks for sharing @gerbermichi . Before I dive in you'll have to excuse the skepticism. If it was this straightforward for images to be created for ARM64 architectures, is there any reason why confluent haven't done it yet?

I was under the impression they had to make significant changes to the source in order to get it to work for ARM64.

[gerbermichi]

I found a solution to build the Kafka image for other CPU architectures, such as ARM64 CPUs. https://nxt.engineering/blog/kafka-docker-image

Thanks for sharing @gerbermichi . Before I dive in you'll have to excuse the skepticism. If it was this straightforward for images to be created for ARM64 architectures, is there any reason why confluent haven't done it yet?

I was under the impression they had to make significant changes to the source in order to get it to work for ARM64.

I understand your skepticism. I also thought at first that it can't be that "simple", however the Docker images run smoothly on my M1.

[kitos9112]

I found a solution to build the Kafka image for other CPU architectures, such as ARM64 CPUs. https://nxt.engineering/blog/kafka-docker-image

Thanks for sharing @gerbermichi . Before I dive in you'll have to excuse the skepticism. If it was this straightforward for images to be created for ARM64 architectures, is there any reason why confluent haven't done it yet? I was under the impression they had to make significant changes to the source in order to get it to work for ARM64.

I understand your skepticism. I also thought at first that it can't be that "simple", however the Docker images run smoothly on my M1.

That sounds quite promising indeed. A developer should not aim for resilience or production-support levels on their own environment or laptop as long as it remains functional and almost close to x64 counterparts.

As long as your code gets compiled and tested in an x86 environment I think this is quite appropriate to fill the current gap

[mikebell90]

I found a solution to build the Kafka image for other CPU architectures, such as ARM64 CPUs. https://nxt.engineering/blog/kafka-docker-image

Thanks for sharing @gerbermichi . Before I dive in you'll have to excuse the skepticism. If it was this straightforward for images to be created for ARM64 architectures, is there any reason why confluent haven't done it yet? I was under the impression they had to make significant changes to the source in order to get it to work for ARM64.

I understand your skepticism. I also thought at first that it can't be that "simple", however the Docker images run smoothly on my M1.

You probably don't use kafka streams.These use native code (RocksDB) and last time I checked this had not been integrated into kafka in an m1 version

[YuniorGlez]

Its not the best solution, but maybe i can help somebody with too many headaches.

You can run the docker build command with the flag --platform and maybe its works. It works for me :)

docker build --platform linux/amd64 .

[andrewegel]

You can run the docker build command with the flag --platform and maybe its works. It works for me :)

Thats for buildx features in Docker Desktop for mac - The issue is that this is a maven project that builds docker images directly through docker's API - Currently there is no "buildx" feature in the maven-dockerfile plugin that we use, nor will there be since its a deprecated project (https://github.com/spotify/dockerfile-maven). So without refactoring all of the build framework and code, the path of solution we have to take is to run the mvn builds on a native ARM64 system.

Up-heaving that build framework back to the way we were doing things in cp-docker-images would undo two years of work/progress to get docker builds into this format. At Confluent we have our reasons for choosing this process through the maven plugin, rather than direct docker buildx invocations and for the time being the business has elected to stick with the maven plugin & take the route of building on ARM64 systems, instead of doing a large rewrite (because there are higher priories / features than doing a full rewrite).

Please remain patient folks - The solution presented in https://nxt.engineering/blog/kafka-docker-image is the route that we're taking, and is what I've been mentioning as a work around for a while now.

[niciqy]

Just in case someone is looking confluent images that actually works on mac M1, I have recompiled them based on https://nxt.engineering/blog/kafka-docker-image and pushed it to docker hub. Use at your own risk. Enjoy!

• niciqy/cp-zookeeper-arm64
• niciqy/cp-kafka-arm64
• niciqy/cp-schema-registry-arm64
• niciqy/cp-kafka-connect-arm64

[anuragagarwal561994]

I have an open source project where I am recompiling and creating arm64 compatible images for open source projects.

I have covered a lot of confluent related projects https://github.com/arm64-compat/confluent-platform

The images are provided in 2 flavours:

• amd64
• arm64

So one need not change it for local setup.

Popular images are provided and will be helpful to everyone till confluent comes up with official release:

• cp-zookeeper
• cp-kafka
• cp-schema-registry
• cp-ksql-server
• cp-kafka-rest

etc.

I have tested these images on my M1 mac and they are working just fine, but can't guarantee for production use case, so use only for local development.

[justinrmiller]

@anuragagarwal561994 Thank you so much for this. We're in the process of moving our entire engineering staff over to M1 macs (about halfway there, and all new hires get M1s) and these images are substantially faster than the images provided by Confluent (which have to be emulated).

[anuragagarwal561994]

Thanks @justinrmiller happy to help :)

[ohookins]

Really need this as M1 macs are becoming the norm.

[rohit2b]

Hi folks,

Confluent has released Docker images for arm64 for development and testing purposes. Check it out across different Confluent Platform components in Docker Hub: https://hub.docker.com/u/confluentinc

For example, here's how to pull the latest CP Kafka image:

docker pull confluentinc/cp-kafka:latest.arm64