search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.26k stars 656 forks source link

No traffic analyze on kafka interface #8425

Closed atemix closed 5 months ago

atemix commented 5 months ago

Ubuntu 22.04.3 LTS ntopng v.6.0.240531

Hello! Ntopng doesn't analyze traffic and networks on kafka interface. Kafka interface is the only one configured. No traffic breakdown, no networks statistics. Local / remote hosts correctly displayed in flow page. Local-networks configured as --local-networks=/etc/ntopng/local-networks.conf No erros / warnings on startup in log. It is the same ntopng instance as in #8409.

ntopng-traffic-1

ntopng-traffic-2

atemix commented 5 months ago

Local HTTP Servers also empty.

lucaderi commented 5 months ago

Can you please paste the nprobe and ntopng configuration files?

atemix commented 5 months ago

Hi @lucaderi

--interface="kafka://127.0.0.1:[XXX]" --instance-name=XXX --data-dir=/[XXX]/ntopng --pcap-dir=/[XXX]/n2disk --http-port=0 --https-port=443 --max-num-hosts=1048576 --max-num-flows=67108864 --dns-mode=3 --local-networks=/etc/ntopng/local-networks.conf --dump-flows="clickhouse;XXX;ntopng;XXX;XXX" --ignore-vlans --offline

--disable-startup-checks= --pid-file=/var/run/nprobe-netflow.pid --collector-port=2055 --kafka="[XXX]:[XXX];flow;option" --kafka-conf="security.protocol=ssl" --kafka-conf="ssl.ca.location=/etc/nprobe/[XXX]" --kafka-conf="ssl.keystore.location=/etc/nprobe/[XXX].jks" --kafka-conf="ssl.keystore.password=[XXX]" -T="@NTOPNG@" -E=0:999 -i=none -n=none -a= --flow-deduplication=15

atemix commented 5 months ago

After updating to dev 6.1.240603 nothing change. Still no traffic breakdown and networks.

lucaderi commented 5 months ago

Please see https://www.ntop.org/nprobe/howto-use-kafka-instead-of-zmq-for-reliable-flow-collection-and-ipc/ that shows how to configure ntopng and nProbe when using Kafka. Your nprobe kafka configuration is good for exporting data to a kafka broker with consumers different from ntopng. If you want to use ntopng you need to do something like