search

dmachard / go-dnscollector

Ingesting, pipelining, and enhancing your DNS logs with usage indicators, security analysis, and additional metadata.
MIT License
213 stars 48 forks source link

does kafka logger support multi address? #679

Open LamberMa opened 7 months ago

LamberMa commented 7 months ago

Is your feature request related to a problem? Please describe. I have a Kafka cluster with three machines, but I haven't found a configuration method for multiple machines in this project. Does this mean that the configuration for multiple hosts is temporarily not supported?

My config as below

    - name: kafka
      kafkaproducer:
        remote-address: kafka1,kafka2,kafka3
        remote-port: 9092

the errors is

Apr 15 18:46:41 office-dns-recursor-vm-172-22-235-188.bj-y-kwai.it go-dnscollector[28107]: INFO: 2024/04/15 18:46:41.153137 logger - [kafka] kafka - retry to connect in 10 seconds
Apr 15 18:46:41 office-dns-recursor-vm-172-22-235-188.bj-y-kwai.it go-dnscollector[28107]: ERROR: 2024/04/15 18:46:41.153156 logger - [kafka] kafka - failed to established connection

The second question is, for my Kafka topic with 30 partitions, I noticed there's a configuration option specifying the partition to write to. If it's not specified, will the data be randomly written to one of these 30 partitions?

thank you ~

dmachard commented 4 months ago

The second question is, for my Kafka topic with 30 partitions, I noticed there's a configuration option specifying the partition to write to. If it's not specified, will the data be randomly written to one of these 30 partitions?

The next release will support writing in round robin way when the partition setting is null.

dmachard commented 4 months ago

I have a Kafka cluster with three machines, but I haven't found a configuration method for multiple machines in this project. Does this mean that the configuration for multiple hosts is temporarily not supported?

Currently, multiple addresses is not supported.