search

ewolff / microservice-kafka

Demo to show how Apache Kafka can be used for communication between microservices
Apache License 2.0
584 stars 302 forks source link

Race condition of topic creation from KAFKA_CREATE_TOPICS & the Spring apps (?) #23

Closed parawanderer closed 1 year ago

parawanderer commented 2 years ago

I was playing around with the demo, adding a third service and trying to scale the Spring app consumers and encountered the issue of for some reason getting stuck at having one partition, in spite of not modifying the docker-compose.yaml file KAFKA_CREATE_TOPICS from the original

Topic: order    PartitionCount: 1       ReplicationFactor: 1    Configs: 
        Topic: order    Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001

Though when I ran the app for the first time, I had no problems having the partitions be created correctly (and then scaling the consumers). I did some searching and found this issue, which seems to have the same issue: https://github.com/wurstmeister/kafka-docker/issues/490

However the proposed solution there did not fix the issue for me in and of itself. I then found this topic https://github.com/wurstmeister/kafka-docker/issues/569 which gave me a solution: disabling KAFKA_AUTO_CREATE_TOPICS_ENABLE. This solution seems to have permanently fixed the partition count issue for me.

(disclaimer: first time working with Kafka)

EDIT: I had some bugs in my extra Spring microservice code concerning json serialisation, without any particular startup error logs regarding this. Poking around in the app after the fix https://github.com/ewolff/microservice-kafka/pull/24 I noticed this. ~Maybe~ probably that was the cause for allowing the race condition to occur.

ewolff commented 1 year ago

Thanks. I suspect your solutions solves the problem. So I will close this issue for the time being.