Move to Apache Kafka container image for testing

[gunnarmorling]

Description

The currently used Red Panda image isn't ideal in terms of similarity of tech used for dev/testing and prod for most people. As discussed with @cescoffier, I've created a container image that runs both Apache Kafka and ZooKeeper, starts reasonably fast and can be used with the Kafka dev service in Quarkus. I'll be happy to follow through with the actual contribution here, provided some guidance.

Plan

• Find / Define a proper image that provides a Kafka broker and start in a reasonable time (~ 5 seconds on most machines). Kraft is a very interesting approach that would avoid having to start Zookeeper
• Add a flag in the Kafka client extension to select the image to use
• Progressively switch to that new image

[quarkus-bot[bot]]

/cc @cescoffier, @ozangunalp

[gunnarmorling]

@cescoffier, how should I proceed to move an image into Quarkus' quay.io org? E.g. where's the source repo for the GraalVM builder image?

[cescoffier]

@gunnarmorling The images are built with CEKit and the modules and assembly are hosted on https://github.com/quarkusio/quarkus-images.

I'm wondering if we don't need another repo for the Kafka one. Maintaining the quarkus-images is already not "easy". Having weekly deployment and graalvm based updates are irrelevant for Kafka.

[gsmet]

I think we need a separate repo. We also need to decide who is going to maintain these images long term.

Because right now, we aren't maintaining them ourselves.

[gunnarmorling]

That's interesting, who is maintaining e.g. the Quarkus builder image currently?

To clarify what this is about, here's the image I've created: https://github.com/gunnarmorling/kafka-testing. I expect effort for this to be really low: it's based on UBI, and once there's support for Java 17 (LTS), the only thing needing updating every now and then would be the Kafka version itself. The image should be re-built and re-published regularly, so to apply the latest OS updates.

Having weekly deployment and graalvm based updates are irrelevant for Kafka.

You're much closer to these details than I am, but wouldn't it be simplest to simply re-publish the Kafka image as part of that process? E.g. same version, latest updates are applied, etc.?

[cescoffier]

We are maintaining the Quarkus builder image. We even have a few tests verifying the behavior. But it still requires some maintenance work (GraalVM updates, dependency changes, base image update...).

For the Kafka image, I believe that a weekly rebuild (pulling the latest ubi image) would be enough. Do you use ubi-java or ubi and install java yourselves? In the later case, we would need to also verify JVM updates.

[cescoffier]

Forgot one important part. For this image, no need to switch to Cekit. There is no module to reuse.

[gunnarmorling]

I believe that a weekly rebuild (pulling the latest ubi image) would be enough

+1

Do you use ubi-java or ubi and install java yourselves?

I need to correct myself, currently it's not using UBI but Fedora. I'm installing the java-16-openjdk-headless package there. I don't think UBI has Java 16 yet, and I wanted to use it to benefit from fast start-up thanks to CDS.

[gunnarmorling]

@cescoffier, so how should we move forward here? Are you going to create a new repo? Or, if not, should I create a PR against the existing one?

[cescoffier]

@gunnarmorling https://github.com/quarkusio/quarkus-kafka-broker-image and you got invited :-D

[gunnarmorling]

Thanks, accepted; will send a PR for the initial contents in a bit.

[loicmathieu]

Hi, We currently didn't use Kafka devservices because it's redpanda so we cannot use it OOTB with schema registry for eg (didn't remember why).

We use the confluentinc/cp-kafka container with testcontainer's KafkaContainer support with embedded Zookeeper, not ideal but not that bad.

[cescoffier]

We are working on an alternative using Kraft. It should land soon.

[cescoffier]

@ozangunalp should we close this one?

[ozangunalp]

Yes, this is fixed with #23340. Further improvements are on the way to reducing the Strimzi startup time using CDS.