search

Axual / ksml

Kafka Streams for Low Code Environments
Apache License 2.0
25 stars 9 forks source link

Example 10-demo-queryabletable is missing a topic #41

Closed tonvanbart closed 2 years ago

tonvanbart commented 2 years ago

Upon building the runner image and running 10-demo-queryabletable as per the instructions in the README, the run fails with the following stacktrace:

2022-05-13T15:35:20,452Z [system] [io.ksml.example.processor-eccaa6cf-4139-49ea-ad4b-33ba260763c3-StreamThread-1] ERROR o.a.k.c.c.i.ConsumerCoordinator -
                [Consumer clientId=io.ksml.example.processor-eccaa6cf-4139-49ea-ad4b-33ba260763c3-StreamThread-1-consumer, groupId=io.ksml.example.processor] User provided listener org.apache.kafka.streams.processor.internals.StreamsRebalanceListener failed on invocation of onPartitionsAssigned for partitions []
org.apache.kafka.streams.errors.MissingSourceTopicException: One or more source topics were missing during rebalance
    at org.apache.kafka.streams.processor.internals.StreamsRebalanceListener.onPartitionsAssigned(StreamsRebalanceListener.java:58)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.invokePartitionsAssigned(ConsumerCoordinator.java:293)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:430)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:449)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:365)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:508)
    at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1261)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:925)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollPhase(StreamThread.java:885)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:720)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:583)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:556)
2022-05-13T15:35:20,455Z [system] [io.ksml.example.processor-eccaa6cf-4139-49ea-ad4b-33ba260763c3-StreamThread-1] ERROR o.apache.kafka.streams.KafkaStreams -
                stream-client [io.ksml.example.processor-eccaa6cf-4139-49ea-ad4b-33ba260763c3] Encountered the following exception during processing and the registered exception handler opted to SHUTDOWN_CLIENT. The streams client is going to shut down now.

Looking in the yaml file, I suspect that topic ksml_sensordata_table is missing.

tonvanbart commented 2 years ago

I tried to add creation of the table topic in the kafka-seup container in docker-compose.yml (which creates all the topic when the compose starts up), however the same stacktrace occurs. Above the stacktrace the following messages appear in the log:

                [Consumer clientId=io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1-consumer, groupId=io.ksml.example.processor] Error while fetching metadata with correlation id 7 : {ksml_sensordata_table=UNKNOWN_TOPIC_OR_PARTITION}
2022-05-13T15:42:25,190Z [system] [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1] ERROR o.a.k.s.p.i.RepartitionTopics -
                stream-thread [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1-consumer] The following source topics are missing/unknown: [ksml_sensordata_table]. Please make sure all source topics have been pre-created before starting the Streams application.
2022-05-13T15:42:25,190Z [system] [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1] ERROR o.a.k.s.p.i.RepartitionTopics -
                stream-thread [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1-consumer] The following source topics are missing/unknown: [ksml_sensordata_table]. Please make sure all source topics have been pre-created before starting the Streams application.
2022-05-13T15:42:25,195Z [system] [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1] ERROR o.a.k.s.p.i.StreamsPartitionAssignor -
                stream-thread [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1-consumer] Caught an error in the task assignment. Returning an error assignment.
org.apache.kafka.streams.errors.MissingSourceTopicException: Missing source topics.

Note: example-producer has a depends_on on kafka-setup, however this does not guarantee that all topics will be created before the producer starts producing (as soon as the container is considered "ready", producing will start).

tonvanbart commented 2 years ago

After having all images rebuilt when doing docker compose up, the offending example also works. Apparently it was still using an old version of the kafka-setup container somehow and the table was not created (?) Will create a PR.