no rrd files created in /var/tmp/ntopng/{IFACE-ID}/rrd/{IP}

[ol3k]

Hi,

I recently installed ntopng with nprobe.

cat /etc/ntopng/ntopng.conf | grep -v "#" -G=/var/run/ntopng.pid -e= --interface="tcp://127.0.0.1:5556" -w=3000 -n=3 -d=/var/tmp/ntopng -q= --community=

cat /etc/nprobe/nprobe-none.conf --zmq="tcp://*:5556" --collector-port=2055 -n=none -i=none

It's looking fine but there are no rrd files for each host and the protocol statistics are showing an error.

drwx------ 4 nobody nogroup 4.0K Jun 22 12:44 . drwx------ 11 nobody nogroup 4.0K Jun 22 12:19 .. -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Amazon.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Apple.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:14 BitTorrent.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:40 Chat.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Citrix.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Cloud.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Cloudflare.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Collaborative.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 DNS.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:40 Database.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Download-FileTransfer-FileSharing.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Dropbox.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:42 Email.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 FTP_CONTROL.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:40 Facebook.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Github.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Google.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 11:30 H323.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 HTTP.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 HTTP_Proxy.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:14 ICMP.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:42 IMAP.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:40 IMAPS.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 MS_OneDrive.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Media.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:40 MySQL.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Network.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:42 Office365.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:14 POP3.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:14 POPS.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:14 PostgreSQL.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 RemoteAccess.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 SOCKS.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 SSH.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 SSL.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Skype.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 SocialNetwork.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:14 Telegram.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Twitch.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Twitter.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Unknown.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Unspecified.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 VoIP.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 Web.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:40 WhatsApp.rrd -rw-r--r-- 1 nobody nogroup 1.1M Jun 22 12:44 bytes.rrd drwx------ 2 nobody nogroup 4.0K Jun 22 11:07 localstats drwx------ 3 nobody nogroup 4.0K Jun 22 11:07 macs -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 num_devices.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 num_flows.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 num_hosts.rrd -rw-r--r-- 1 nobody nogroup 31K Jun 22 12:44 num_http_hosts.rrd -rw-r--r-- 1 nobody nogroup 695K Jun 22 12:44 num_zmq_rcvd_flows.rrd -rw-r--r-- 1 nobody nogroup 1.1M Jun 22 12:44 packets.rrd -rw-r--r-- 1 nobody nogroup 369K Jun 22 12:44 tcp_finack.rrd -rw-r--r-- 1 nobody nogroup 369K Jun 22 12:44 tcp_lost.rrd -rw-r--r-- 1 nobody nogroup 369K Jun 22 12:44 tcp_ooo.rrd -rw-r--r-- 1 nobody nogroup 369K Jun 22 12:44 tcp_retransmissions.rrd -rw-r--r-- 1 nobody nogroup 369K Jun 22 12:44 tcp_rst.rrd -rw-r--r-- 1 nobody nogroup 369K Jun 22 12:44 tcp_syn.rrd -rw-r--r-- 1 nobody nogroup 369K Jun 22 12:44 tcp_synack.rrd

nobody 7789 0.7 0.5 497216 20552 ? Ssl 12:22 0:12 /usr/local/bin/nprobe /run/nprobe.conf

nobody 8094 12.4 2.7 1499224 111444 ? Ssl 12:42 0:52 /usr/local/bin/ntopng /run/ntopng.conf

tcp 0 0 127.0.0.1:5556 0.0.0.0: LISTEN 0 1079127 7789/nprobe tcp 0 0 0.0.0.0:3000 0.0.0.0: LISTEN 0 1084106 8094/ntopng tcp 0 0 127.0.0.1:3306 0.0.0.0: LISTEN 112 1018882 710/mysqld tcp 0 0 127.0.0.1:6379 0.0.0.0: LISTEN 111 243260 18088/redis-server udp 0 0 0.0.0.0:2055 0.0.0.0: 65534 1079132 7789/nprobe udp6 0 0 :::2055 ::: 65534 1079133 7789/nprobe

What is wrong with this config? Thanks.

[emanuele-f]

Hello, what ntopng version are you using? Please run ntopng --version. Have you enabled the per host timeseries from the ntopng preferences?

[ol3k]

Hi @emanuele-f

v.3.4.180621 [Enterprise/Professional build] GIT rev: 3.4-stable:fa4615f95326ff3db4ca5f59d2cee0f113a2d2d3:20180621 Pro rev: r1595 Built on: Debian GNU/Linux 9.1 (stretch) System Id: 68878E8276066B21 Platform: x86_64 Edition: Enterprise License Type: Demo

I enabled anthing in timeseries for testing purposes.

[emanuele-f]

I think you are missing the -m option. Since you are capturing from nprobe, you have to tell ntopng which are your local networks. For example -m="192.168.1.0/24,10.1.0.0/16". After this, a chart icon will happear under your local hosts.

[ol3k]

Yes, thanks. That was missing. Now the chart icon and RRD files are available.

Another question not related to rrd, but I didn't find a solution yet:

Is there any possibility to just gather flows of specific nets in the collector modes? The --bpf-filter isn't working with ntopng or nprobe in collector-mode, the --collection-filter in nprobe seems to have no effect.

EDIT: after some reading: I think this could be addressed to https://github.com/ntop/ntopng/issues/1782 I will update to latest unstable version and report again later.

EDIT2: It seems that the nprobe collection-filter is still not working: nProbe v.8.5.180626 ($Revision: 6186 $) for x86_64-pc-linux-gnu with native PF_RING acceleration

nprobe -i none -n none --collection-filter "!172.16.0.0/12" -3 2055 --zmq tcp://127.0.0.1:5556

26/Jun/2018 13:58:27 [plugin.c:179] No plugins found in ./plugins 26/Jun/2018 13:58:27 [plugin.c:187] Loading 24 plugins [.so] from /usr/local/lib/nprobe/plugins 26/Jun/2018 13:58:27 [nprobe.c:3949] ERROR: Invalid nProbe license (/etc/nprobe.license) [Missing license file] 26/Jun/2018 13:58:27 [nprobe.c:3956] ERROR: 26/Jun/2018 13:58:27 [nprobe.c:3957] ERROR: 26/Jun/2018 13:58:27 [nprobe.c:3958] ERROR: Switching to DEMO MODE (missing valid license) 26/Jun/2018 13:58:27 [nprobe.c:3959] ERROR: 26/Jun/2018 13:58:27 [nprobe.c:3960] ERROR: Purchase your nProbe license at 26/Jun/2018 13:58:27 [nprobe.c:3961] ERROR: https://shop.ntop.org/ 26/Jun/2018 13:58:27 [nprobe.c:3962] ERROR: 26/Jun/2018 13:58:27 [nprobe.c:3963] ERROR: 26/Jun/2018 13:58:27 [nprobe.c:5628] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 26/Jun/2018 13:58:27 [nprobe.c:5631] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 26/Jun/2018 13:58:27 [nprobe.c:5718] Welcome to nProbe v.8.5.180626 ($Revision: 6186 $) for x86_64-pc-linux-gnu with native PF_RING acceleration 26/Jun/2018 13:58:27 [nprobe.c:5728] Running on Debian GNU/Linux 9.1 (stretch) 26/Jun/2018 13:58:27 [nprobe.c:5739] [LICENSE] nProbe SystemId: 68878E8276066B21 26/Jun/2018 13:58:27 [nprobe.c:5806] Sample rate [packet: 1][flow collection/export: 1/1] 26/Jun/2018 13:58:27 [nprobe.c:8443] ERROR: 26/Jun/2018 13:58:27 [nprobe.c:8444] ERROR: NOTE: This is a DEMO version limited to 25000 flows export. 26/Jun/2018 13:58:27 [nprobe.c:8445] ERROR: 26/Jun/2018 13:58:27 [modbusPlugin.c:100] [MODBUS] Idle flow timeout set to 120 sec 26/Jun/2018 13:58:27 [nprobe.c:8451] Welcome to nProbe v.8.5.180626 for x86_64-pc-linux-gnu 26/Jun/2018 13:58:27 [nprobe.c:7366] WARNING: Adding %EXPORTER_IPV4_ADDRESS to the template as nProbe is working as collector 26/Jun/2018 13:58:27 [plugin.c:1235] 0 plugin(s) enabled 26/Jun/2018 13:58:27 [nprobe.c:7936] Non IPv4/v6 traffic is discarded according to the template 26/Jun/2018 13:58:27 [util.c:465] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat 26/Jun/2018 13:58:27 [util.c:476] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat 26/Jun/2018 13:58:27 [nprobe.c:8656] IPv6 traffic will NOT be exported/accounted by this probe 26/Jun/2018 13:58:27 [nprobe.c:8657] due to configuration options (e.g. use NetFlow v9) 26/Jun/2018 13:58:27 [nprobe.c:8658] Please use -V to set the version to other than NetFlow V5 26/Jun/2018 13:58:27 [nprobe.c:8809] Not capturing packet from interface (collector mode) 26/Jun/2018 13:58:27 [util.c:4566] Initializing ZMQ as server 26/Jun/2018 13:58:27 [util.c:4609] Succesfully created ZMQ endpoint tcp://127.0.0.1:5556 26/Jun/2018 13:58:27 [util.c:3644] nProbe changed user to 'nobody' 26/Jun/2018 13:58:27 [collect.c:144] Flow collector listening on port 2055 (IPv4/v6) 26/Jun/2018 13:58:27 [nprobe.c:9055] nProbe started successfully

Am I correct that we should see with e.g.:

nprobe -i none -n none --collection-filter "192.168.1.1" -3 2055 --zmq tcp://127.0.0.1:5556

just flows with source/destination 192.168.1.1?

[emanuele-f]

Which is the collection filter you are using? The fix for that issue has been backported to stable, so if you have an updated nprobe it should work.

[ol3k]

I try to filter a single proxy ip. In production environment there should be about 12 of /24s.

grep -v "#" /etc/nprobe/nprobe.conf -i=none -n=none --collection-filter="172.16.0.180/32" -3=2055 --zmq=tcp://127.0.0.1:5556

[emanuele-f]

I've just tested the collection filter and it work properly for me. The filter above tells nprobe to only export traffic which has source IP or destination IP 172.16.0.180 .

Please note that you are using nprobe 8.5 (dev) with ntopng 3.4 (stable) and this is not recommended. Please stick with dev or stable releases only.

[ol3k]

hmm that's bad :-(

During the nprobe update I updated ntopng, too.

This should be fine:

Welcome to ntopng x86_64 v.3.5.180626

Welcome to nProbe v.8.5.180626 ($Revision: 6186 $)

Any suggestions I can try?

[emanuele-f]

That versions are ok now.

If you can provide a pcap file with the netflow traffic as seen by nprobe I can perform tests with your exact flows. If so, please send me a pcap at faranda@ntop.org . You can use tcpdump -i eth0 -s0 -w netflow.pcap where eth0 is your nic to generate it. Please also use wireshark to verify that the pcap contains both netflow template and data.

[ol3k]

I should capture on the listening port of nprobe the switch sends its data to?

0.0.0.0:2055 0.0.0.0:* 65534 2519954 11226/nprobe

btw: I played with the "--black-list" setting, it seems to work as expected. But I can't filter the needed /24s by blacklisting all others... I guess I need a working "--collection-filter"

[emanuele-f]

Yes, something similar to tcpdump -i eth0 -s0 -w netflow.pcap udp port 2055

[emanuele-f]

So sFlow is being used, not netflow. I've reproduced your issue, I've opened a bug. Please follow https://github.com/ntop/nProbe/issues/275 .