BrokerNotAvailableError: Broker not available

xzmagic commented 6 years ago

Questions?

Check the Troubleshooting / FAQ ?
Search the issues ?

Bug Report

when send data to kafka, get err : BrokerNotAvailableError: Broker not available ,end then process exit() and can't catch excetipion description for this unexpected behaviour, even i add code below: process.on('uncaughtException', function (err) { console.log('Caught exception: ' + err.message + " " + err.stack); });

Environment

Node version:v5.2
Kafka-node version:2.6.1
Kafka version: 2.2.1

For specific cases also provide

Number of Brokers:
Number partitions for topic:

Include Sample Code to reproduce behavior

// include code here
kafkaPush.prototype.reconnect = function () {
    this_ = this;
    this_.kafkaIsReady = false;
    this_.client.close();

    if (typeof this_.producer.close === 'function') {
        console.log("\x1b[32m call producer close\x1b[0m");
        this_.producer.close();
        console.log("\x1b[32m producer close done\x1b[0m");
    }
    if (this_.reconnectTimer == null) { // Multiple Error Events may fire, only set one connection retry.
        this_.reconnectTimer = setTimeout(function () {
            this_.setupConnectionToKafka();
        }, 3000);
    }
};

Include output with Debug turned on

:BrokerNotAvailableError: Broker not available

Thanks for your contribution!

manesioz commented 4 years ago

Any updates on this? Seems like many people are facing this issue (including myself). Duplicate: https://github.com/SOHU-Co/kafka-node/issues/1016

jaiswarvipin commented 4 years ago

My docker-compose file

version: '3'
services:
  zookeeper:
    container_name: zookeeper
    image: confluentinc/cp-zookeeper
    ports:
      - "32181:32181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2

  kafka:
    container_name: kafka
    image: confluentinc/cp-kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: localhost:32181
      KAFKA_LISTENERS: INTERNAL://localhost:9092,OUTSIDE://localhost:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://localhost:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      ES_JAVA_OPTS: "-Xms512m -Xmx3000m"

and Producer code is

var kafka = require('kafka-node'),
    Producer = kafka.Producer,
    KeyedMessage = kafka.KeyedMessage,
    client = new kafka.KafkaClient({kafkaHost:"localhost:9094"}),

    producer = new Producer(client),
    km = new KeyedMessage('key', 'message'),
    payloads = [
        { topic: 'topic1', messages: 'hi', partition: 0 },
        { topic: 'topic1', messages: ['hello', 'world', km] }
    ];

client.createTopics(topicsToCreate, (error, result) => {

                echo ("------------------------KAFAK--------------------")
                console.log(error);
                console.log(result);            
            });

Getting below error, while creating the topic before sending pay-load to topics

{ BrokerNotAvailableError: Broker not available (loadMetadataForTopics)
    at new BrokerNotAvailableError (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\errors\BrokerNotAvailableError.js:11:9)
    at KafkaClient.loadMetadataForTopics (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:891:21)
    at KafkaClient.loadMetadata (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:876:8)
    at KafkaClient.getController (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:267:8)
    at KafkaClient.sendControllerRequest (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:1219:8)
    at KafkaClient.createTopics (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:935:8)
    at C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\index_behind_kong.js:60:11
    at C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\baseClient.js:370:18
    at KafkaClient.loadMetadataForTopics (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:891:12)
    at RetryOperation._fn (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\baseClient.js:360:12) message: 'Broker not available (loadMetadataForTopics)' }
undefined

I have gave the delay of 5 seconds before calling to createtopic method, however no luck found.

Kindly assist.

jaiswarvipin commented 4 years ago

Finally got the solution. I was doing on mistake. By default Apache Kafka create the 3 replication, this mean its by default create 3 broker as well. However aboe YAML create only one broker, and looknig for other 2 which is not created. hence we are getting the same error.

Fix

version: '3'
services:
  zookeeper:
    container_name: zookeeper
    image: confluentinc/cp-zookeeper
    ports:
      - "32181:32181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2

  kafka:
    container_name: kafka
    image: confluentinc/cp-kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
      KAFKA_LISTENERS: INTERNAL://:9092,OUTSIDE://:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      ES_JAVA_OPTS: "-Xms512m -Xmx3000m"

then KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 tell the Apache kafka that we Wanted to create only one replication and its work

Happy coding :)

Thanks & Regards Jaiswar Vipin Kumar R.

SOHU-Co / kafka-node