search

apache / camel-quarkus

Apache Camel Quarkus
https://camel.apache.org
Apache License 2.0
251 stars 186 forks source link

Intermittent failure of Kafka example project container startup #6171

Open jamesnetherton opened 1 month ago

jamesnetherton commented 1 month ago

Bug description

This seems to pop up on a regular basis for the example projects build jobs.

Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/strimzi-test-container/test-container:latest-kafka-3.2.1
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:359)
    at io.strimzi.test.container.StrimziKafkaContainer.doStart(StrimziKafkaContainer.java:109)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:330)
    at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.lambda$startKafka$5(DevServicesKafkaProcessor.java:258)
    at java.base/java.util.Optional.orElseGet(Optional.java:364)
    at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.startKafka(DevServicesKafkaProcessor.java:285)
    at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.startKafkaDevService(DevServicesKafkaProcessor.java:95)
    ... 12 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:344)
    ... 18 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:563)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:354)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
    ... 19 more
Caused by: java.lang.IllegalStateException: Wait strategy failed. Container exited with code 1
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:533)
    ... 21 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*Transitioning from RECOVERY to RUNNING.*'
    at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
    at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
    at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:909)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:500)
    ... 21 more
jamesnetherton commented 1 month ago

I've also seen this happen for the tests in the Kafka extension deployment module.

jamesnetherton commented 1 month ago

Some more info from the logs:

2024-06-10 09:17:58,108 ERROR [tc.qua.io/.2.1] (build-7) Log output from the failed container:
usage: kafka-storage format [-h] --config CONFIG --cluster-id CLUSTER_ID
                     [--ignore-formatted]
kafka-storage: error: argument --cluster-id/-t: expected one argument
[2024-06-10 09:17:00,484] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
okeeper.common.X509Util)
[2024-06-10 09:17:00,859] WARN No meta.properties file under dir /tmp/kraft-combined-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2024-06-10 09:17:00,861] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
kafka.common.KafkaException: No `meta.properties` found in /tmp/kraft-combined-logs (have you run `kafka-storage.sh` to format the directory?)
    at kafka.server.BrokerMetadataCheckpoint$.$anonfun$getBrokerMetadataAndOfflineDirs$2(BrokerMetadataCheckpoint.scala:171)
    at scala.collection.immutable.List.foreach(List.scala:333)
    at kafka.server.BrokerMetadataCheckpoint$.getBrokerMetadataAndOfflineDirs(BrokerMetadataCheckpoint.scala:160)
LogDirs(KafkaRaftServer.scala:155)
    at kafka.server.KafkaRaftServer.<init>(KafkaRaftServer.scala:56)
    at kafka.Kafka$.buildServer(Kafka.scala:79)
    at kafka.Kafka$.main(Kafka.scala:87)
    at kafka.Kafka.main(Kafka.scala)