Kafka Dev Services fails to start since 2.16.0

[jendib]

Describe the bug

I'm getting this error since I upgraded to 2.16.0 when running Kafka Dev Services on my Gitlab CI. Please tell me if you need a reproducer.

2023-01-30 19:24:59,001 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pulling image layers:  2 pending,  5 downloaded,  0 extracted, (61 MB/? MB)
2023-01-30 19:24:59,608 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pulling image layers:  1 pending,  6 downloaded,  0 extracted, (113 MB/? MB)
2023-01-30 19:24:59,749 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pulling image layers:  0 pending,  7 downloaded,  0 extracted, (113 MB/133 MB)
2023-01-30 19:25:00,439 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pulling image layers:  0 pending,  7 downloaded,  1 extracted, (116 MB/133 MB)
2023-01-30 19:25:00,942 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pulling image layers:  0 pending,  7 downloaded,  2 extracted, (118 MB/133 MB)
2023-01-30 19:25:01,204 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pulling image layers:  0 pending,  7 downloaded,  3 extracted, (118 MB/133 MB)
2023-01-30 19:25:01,490 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pulling image layers:  0 pending,  7 downloaded,  4 extracted, (118 MB/133 MB)
2023-01-30 19:25:03,786 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pulling image layers:  0 pending,  7 downloaded,  5 extracted, (126 MB/133 MB)
2023-01-30 19:25:06,688 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pulling image layers:  0 pending,  7 downloaded,  6 extracted, (131 MB/133 MB)
2023-01-30 19:25:07,259 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pulling image layers:  0 pending,  7 downloaded,  7 extracted, (133 MB/133 MB)
2023-01-30 19:25:07,453 INFO  [🐳 .io/.3.4]] (docker-java-stream--111821119) Pull complete. 7 layers, pulled in 10s (downloaded 133 MB at 13 MB/s)
2023-01-30 19:25:07,456 INFO  [🐳 .io/.3.4]] (build-2) Creating container for image: docker.io/vectorized/redpanda:v22.3.4
2023-01-30 19:25:09,649 INFO  [🐳 .io/.3.4]] (build-2) Container docker.io/vectorized/redpanda:v22.3.4 is starting: 8477ff5e1b1394c6d1f2f062f065b14200bbc7e4af18bbad519ffa1515f938f7
2023-01-30 19:26:10,829 ERROR [🐳 .io/.3.4]] (build-2) Could not start container: java.lang.IllegalStateException: Container exited with code 1
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:514)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
    at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.lambda$startKafka$5(DevServicesKafkaProcessor.java:236)
    at java.base/java.util.Optional.orElseGet(Optional.java:364)
    at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.startKafka(DevServicesKafkaProcessor.java:282)
    at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.startKafkaDevService(DevServicesKafkaProcessor.java:95)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:909)
    at io.quarkus.builder.BuildContext.run(BuildContext.java:281)
    at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
    at java.base/java.lang.Thread.run(Thread.java:833)
    at org.jboss.threads.JBossThread.run(JBossThread.java:501)
2023-01-30 19:26:10,858 ERROR [🐳 .io/.3.4]] (build-2) Log output from the failed container:
WARNING: This is a setup for development purposes only; in this mode your clusters may run unrealistically fast and data can be corrupted any time your computer shuts down uncleanly.
We'd love to hear about your experience with redpanda:
https://redpanda.com/feedback
Starting redpanda...
Running:
HOSTNAME=8477ff5e1b13 EDITOR=vi PWD=/ HOME=/var/lib/redpanda SHLVL=1 PATH=/opt/redpanda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /opt/redpanda/bin/redpanda redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml --memory=1G --overprovisioned --unsafe-bypass-fsync=true --lock-memory=false --reserve-memory=0M --smp=1
Welcome to the Redpanda community!
Documentation: https://docs.redpanda.com/ - Product documentation site
GitHub Discussion: https://github.com/redpanda-data/redpanda/discussions - Longer, more involved discussions
GitHub Issues: https://github.com/redpanda-data/redpanda/issues - Report and track issues with the codebase
Support: https://support.redpanda.com/ - Contact the support team privately
Product Feedback: https://redpanda.com/feedback - Let us know how we can improve your experience
Slack: https://redpanda.com/slack - Chat about all things Redpanda. Join the conversation!
Twitter: https://twitter.com/redpandadata - All the latest Redpanda news!
WARN  2023-01-30 19:25:10,963 [shard 0] seastar - Creation of perf_event based stall detector failed, falling back to posix timer: std::__1::system_error (error system:1, perf_event_open() failed: Operation not permitted)
ERROR 2023-01-30 19:25:10,965 [shard 0] seastar - Operation not permitted

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

Java 17

GraalVM version (if different from Java)

NA

Quarkus version or git rev

2.16.0

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot[bot]]

/cc @alesj (kafka), @cescoffier (kafka), @geoand (devservices), @ozangunalp (kafka), @stuartwdouglas (devservices)

[jendib]

I tried the older redpanda version used in 2.15.x with quarkus.kafka.devservices.image-name=docker.io/vectorized/redpanda:v22.1.7 and it seems to work fine so it may be unrelated to Quarkus. I'm leaving this open if you have anything to comment on that.

[sberyozkin]

I checked 2.16.1 (as part of my demo) and it worked fine

[cescoffier]

@jendib Can you try with Redpanda v22.3.11?

[jendib]

@cescoffier 22.3.11 works but 22.3.4 does not. It's working locally but I still have the same issue on my Gitlab CI runner.

[cescoffier]

Thanks, any detail you can share about your gitlab runner?

There are a few options we can pass to red panda, but it's hard to say which one.

BTW, you can also use the kafka native dev service (a natively compiled kafka broker).

[jendib]

@cescoffier It's a Docker server running Debian. The machine is quite old but other than that it's fairly standard. From what I can find the error comes from a change they made in the underlying database system (scylladb), so I guess there is not much we can do except using an older version or another image as you suggested.

[jesantana]

I had the same Issue running 2.16.1 on a Fedora 32 wit Docker 19

• 22.3.4 didn't work
• 22.3.11 didn't work
• 22.1.7 worked ok

[cescoffier]

Did you open an issue on RedPanda?

[jendib]

I did not.

[sberyozkin]

@jesantana I run the demo on Fedora 37

[cescoffier]

I'm going to close that issue as it's a Red Panada issue, and there is not much we can do on our side. Please open an issue against Red Panda.