Closed tonvanbart closed 2 years ago
I tried to add creation of the table topic in the kafka-seup
container in docker-compose.yml
(which creates all the topic when the compose starts up), however the same stacktrace occurs.
Above the stacktrace the following messages appear in the log:
[Consumer clientId=io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1-consumer, groupId=io.ksml.example.processor] Error while fetching metadata with correlation id 7 : {ksml_sensordata_table=UNKNOWN_TOPIC_OR_PARTITION}
2022-05-13T15:42:25,190Z [system] [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1] ERROR o.a.k.s.p.i.RepartitionTopics -
stream-thread [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1-consumer] The following source topics are missing/unknown: [ksml_sensordata_table]. Please make sure all source topics have been pre-created before starting the Streams application.
2022-05-13T15:42:25,190Z [system] [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1] ERROR o.a.k.s.p.i.RepartitionTopics -
stream-thread [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1-consumer] The following source topics are missing/unknown: [ksml_sensordata_table]. Please make sure all source topics have been pre-created before starting the Streams application.
2022-05-13T15:42:25,195Z [system] [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1] ERROR o.a.k.s.p.i.StreamsPartitionAssignor -
stream-thread [io.ksml.example.processor-feeb64ce-48bb-48c8-bccf-46cfcc8f6054-StreamThread-1-consumer] Caught an error in the task assignment. Returning an error assignment.
org.apache.kafka.streams.errors.MissingSourceTopicException: Missing source topics.
Note: example-producer
has a depends_on
on kafka-setup
, however this does not guarantee that all topics will be created before the producer starts producing (as soon as the container is considered "ready", producing will start).
After having all images rebuilt when doing docker compose up
, the offending example also works. Apparently it was still using an old version of the kafka-setup
container somehow and the table was not created (?)
Will create a PR.
Upon building the runner image and running
10-demo-queryabletable
as per the instructions in the README, the run fails with the following stacktrace:Looking in the yaml file, I suspect that topic
ksml_sensordata_table
is missing.