search

conduktor / kafka-stack-docker-compose

docker compose files to create a fully working kafka stack
Apache License 2.0
3.28k stars 1.27k forks source link

Setup works only in localhost #18

Closed prem911 closed 6 years ago

prem911 commented 6 years ago

When I setup the broker and zookeeper in a remote machine, then any message produced throws this error. Same for single or multi broker/zk arrangement.

$ bin/kafka-console-producer.sh --broker-list server-url:9092,server-url:9093,server-url:9094 --topic test

message1

ERROR Error when sending message to topic test with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-1: 1511 ms has passed since batch creation plus linger time

Note: I have the entry for /etc/hosts for kafka1

simplesteph commented 6 years ago

Hi Prem, Setting up Kafka is hard. This is only intended for local development, and works as intended. This setup won't work in the cloud without substantial investment on your end to understand the setting KAFKA_ADVERTISED_LISTENERS. For short, set it to your remote machine's ip (public if you go over the internet, private if not). Happy learning!

prem911 commented 6 years ago

Thanks Stephane! Realising now that Kafka in blogs and Kafka in practical is way different! I have gone through the various ENV params for external deployment. However, those have not yielded much result. Well, I will kind my way out.

simplesteph commented 6 years ago

Hi @prem911 . I do have a course online dedicated to understanding how to deploy Kafka externally. After looking through it I'm sure you'll understand how to deploy Kafka correctly. https://www.udemy.com/kafka-streams/?couponCode=GITHUB

Happy learning! Stephane