Not able to write output into KAFKA

ACodingfreak commented 8 months ago

Describe the bug

As shown in below logs, I am running goflow2 and kafka containers in 2 separate machines. When I try to write output into a kafka topic on the broker 10.10.26.207:9092 it just fails.

goflow2 is able to create the respective topic in the kafka instance but facing issue when writing the messages into it as it tries to resolve dns for "kafka1" which is the container name which is hosting the kafka instance.

Any specific reason why it tries to pick "kafka1" vs configured broker ?

312:~/goflow2/goflow2$ sudo docker container run -p 6343:6343/udp -p 2055:2055/udp -it --name goflow2 netsampler/goflow2:ee095a9
~ $ 
~ $ 
~ $ ./goflow2 -transport=kafka -transport.kafka.brokers=10.10.26.207:9092 -transport.kafka.topic=flows -format=text
INFO[0000] starting GoFlow2                             
INFO[0000] starting collection                           blocking=false count=1 hostname= port=6343 queue_size=1000000 scheme=sflow workers=2
INFO[0000] starting collection                           blocking=false count=1 hostname= port=2055 queue_size=1000000 scheme=netflow workers=2
ERRO[0077] transport error                               error="kafka transport kafka: Failed to produce message to topic flows: dial tcp: lookup kafka1 on 10.10.13.22:53: server misbehaving"
ERRO[0102] transport error                               error="kafka transport kafka: Failed to produce message to topic flows: dial tcp: lookup kafka1 on 10.10.13.22:53: server misbehaving"
ERRO[0122] transport error                               error="kafka transport kafka: Failed to produce message to topic flows: dial tcp: lookup kafka1 on 10.10.13.22:53: server misbehaving"
ERRO[0142] transport error                               error="kafka transport kafka: Failed to produce message to topic flows: dial tcp: lookup kafka1 on 10.10.13.22:53: server misbehaving"
ERRO[0162] transport error                               error="kafka transport kafka: Failed to produce message to topic flows: dial tcp: lookup kafka1 on 10.10.13.22:53: server misbehaving"

To Reproduce Steps to reproduce the behavior:

Run GoFlow2 with arguments '...' ./goflow2 -transport=kafka -transport.kafka.brokers=10.10.26.207:9092 -transport.kafka.topic=flows -format=text
Receive samples '....'
See error

Expected behavior goflow2 should be able to write into respective topic created in kafka

Captures If applicable, add output (JSON, protobuf), packet captures and device configuration.

Sampler device:

Brand: [e.g. Cisco, Juniper, Arista, Mikrotik]
Model: [e.g. MX, QFX]
Version: [e.g. 22]
Estimated flow traffic: [e.g. 5000 samples per second]

GoFlow2:

Version: [e.g. v2.0.0] -~ $ ./goflow2 -v GoFlow2 v2.1.2-1-gee095a9 (2024-02-27T11:23:15-0800)
Environment: [e.g. Kubernetes, Docker, Debian package] Docker
OS: [e.g. Linux Ubuntu Server 23.04] Ubuntu 22.04

Additional context Add any other context about the problem here.

lspgn commented 7 months ago

Hello @ACodingfreak, Are you running Kafka inside a docker-compose?

My guess is that you do not have Kafka configured with KAFKA_ADVERTISED_LISTENERS=10.10.26.207:9092 which results in Kafka giving its hostname kafka1. You could also manually add the following inside the /etc/hosts file of the machine where GoFlow2 lives:

10.10.26.207 kafka1

ACodingfreak commented 7 months ago

@lspgn - Thanks for the reply

Yes I am running kafka broker via docker-compose in node2 and goflow2 as a container in node1. Below is the sample docker-compose configuration I was using for kafka broker.

      KAFKA_BROKER_ID: 1
      KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL_SAME_HOST://:29092,EXTERNAL_DIFFERENT_HOST://:29093
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://:9092,EXTERNAL_SAME_HOST://localhost:29092,EXTERNAL_DIFFERENT_HOST://:29093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT,EXTERNAL_DIFFERENT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

But when i execute the goflow2 inside a container from node1, I can see that topic named flows is created properly but writing messages into kafka is failing still with the DNS resolution error as shown below

~ $ ./goflow2 -transport=kafka -transport.kafka.brokers=10.10.26.207:29093 -transport.kafka.topic=flows -format=text
INFO[0000] starting GoFlow2
INFO[0000] starting collection                           blocking=false count=1 hostname= port=6343 queue_size=1000000 scheme=sflow workers=2
INFO[0000] starting collection                           blocking=false count=1 hostname= port=2055 queue_size=1000000 scheme=netflow workers=2
ERRO[0083] transport error                               error="kafka transport kafka: Failed to produce message to topic flows: dial tcp: lookup kafka1 on 10.10.13.22:53: server misbehaving"

lspgn commented 7 months ago

@ACodingfreak : yes, the advertised listener is wrong, it keeps returning kafka1:9092 this is why I was also suggesting the /etc/hosts method since I'm not heavily familiar with Kafka's environment variable and it's beyond the scope of GoFlow2.

or try KAFKA_ADVERTISED_LISTENERS: CLIENT://localhost:9092 and make sure the docker-compose open ports 9092

But have a look at the following https://docs.confluent.io/platform/current/kafka/multi-node.html

netsampler / goflow2

Not able to write output into KAFKA #291