search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.15k stars 648 forks source link

No Data Available. #1749

Closed Karl-MCS closed 6 years ago

Karl-MCS commented 6 years ago

Good afternoon. I set up my first nProbe/ntopng virtual server a while back collecting flows from a Cisco ASA, and it's been working fine for months.

I'm setting up a second one at a different site, basically the same setup... but I can only intermittently get ntopng to display any data on the webpage, if ever at all. I was hoping someone would be able to help me figure this out. Here is some info. The system is Ubuntu 16.04.4. I'm testing with these commands and have the same or similar in my config files.

sudo nprobe -n none -i none --collector-port 6343 --zmq "tcp://127.0.0.1:5556" --online-license-check
sudo ntopng -i "tcp://127.0.0.1:5556" -w 80 --local-networks "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" --online-license-check

When I do this, they both seem to start up just fine. (System ID and license omitted)

Here is ntopng:

17/Apr/2018 14:22:15 [Ntop.cpp:1485] Setting local networks to 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 17/Apr/2018 14:22:15 [Redis.cpp:111] Successfully connected to redis 127.0.0.1:6379@0 17/Apr/2018 14:22:15 [Redis.cpp:111] Successfully connected to redis 127.0.0.1:6379@0 17/Apr/2018 14:22:15 [NtopPro.cpp:223] [LICENSE] Reading license from /etc/ntopng.license 17/Apr/2018 14:22:15 [NtopPro.cpp:121] Contacting licensing server. Please hold on... 17/Apr/2018 14:22:16 [NtopPro.cpp:139] [LICENSE] /etc/ntopng.license: found valid Enterprise license 17/Apr/2018 14:22:16 [Ntop.cpp:1612] Registered interface tcp://127.0.0.1:5556 [id: 2] 17/Apr/2018 14:22:16 [main.cpp:301] PID stored in file /var/run/ntopng.pid 17/Apr/2018 14:22:17 [HTTPserver.cpp:841] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL. 17/Apr/2018 14:22:17 [Utils.cpp:457] User changed to nobody 17/Apr/2018 14:22:17 [HTTPserver.cpp:912] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts] 17/Apr/2018 14:22:17 [HTTPserver.cpp:915] HTTP server listening on port(s) 80 17/Apr/2018 14:22:17 [main.cpp:383] Working directory: /var/tmp/ntopng 17/Apr/2018 14:22:17 [main.cpp:385] Scripts/HTML pages directory: /usr/share/ntopng 17/Apr/2018 14:22:17 [Ntop.cpp:385] Welcome to ntopng x86_64 v.3.2.180315 - (C) 1998-17 ntop.org 17/Apr/2018 14:22:17 [Ntop.cpp:395] Built on Ubuntu 16.04.4 LTS 17/Apr/2018 14:22:17 [NtopPro.cpp:464] [LICENSE] System Id:
17/Apr/2018 14:22:17 [NtopPro.cpp:465] [LICENSE] Edition: Enterprise 17/Apr/2018 14:22:17 [NtopPro.cpp:466] [LICENSE] License Type: Permanent License 17/Apr/2018 14:22:17 [NtopPro.cpp:469] [LICENSE] License:
17/Apr/2018 14:22:17 [NtopPro.cpp:479] [LICENSE] Maintenance: Until Thu Feb 28 11:29:20 2019 [316 days left] 17/Apr/2018 14:22:17 [PeriodicActivities.cpp:59] Started periodic activities loop... 17/Apr/2018 14:22:17 [PeriodicActivities.cpp:100] Each periodic activity script will use 2 threads 17/Apr/2018 14:22:17 [NetworkInterface.cpp:2326] Started packet polling on interface tcp://127.0.0.1:5556 [id: 2]... 17/Apr/2018 14:22:18 [CollectorInterface.cpp:122] Collecting flows on tcp://127.0.0.1:5556

And nProbe:

17/Apr/2018 14:38:01 [plugin.c:188] No plugins found in ./plugins 17/Apr/2018 14:38:01 [plugin.c:196] Loading 23 plugins [.so] from /usr/local/lib/nprobe/plugins 17/Apr/2018 14:38:01 [nprobe.c:2289] Contacting licensing server. Please hold on... 17/Apr/2018 14:38:01 [nprobe.c:3801] Valid nProbe Pro license found 17/Apr/2018 14:38:01 [nprobe.c:5758] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 17/Apr/2018 14:38:01 [nprobe.c:5761] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 17/Apr/2018 14:38:01 [nprobe.c:5862] Welcome to nProbe Pro v.8.2.180315 ($Revision: 6003 $) for x86_64-pc-linux-gnu with native PF_RING acceleration 17/Apr/2018 14:38:01 [nprobe.c:5872] Running on Ubuntu 16.04.4 LTS 17/Apr/2018 14:38:01 [nprobe.c:5883] [LICENSE] nProbe SystemId: 17/Apr/2018 14:38:01 [nprobe.c:5996] Sample rate [packet: 1][flow collection/export: 1/1] 17/Apr/2018 14:38:01 [nprobe.c:8443] Welcome to nProbe v.8.2.180315 for x86_64-pc-linux-gnu 17/Apr/2018 14:38:01 [nprobe.c:7367] WARNING: Adding %EXPORTER_IPV4_ADDRESS to the template as nProbe is working as collector 17/Apr/2018 14:38:01 [plugin.c:1156] 0 plugin(s) enabled 17/Apr/2018 14:38:01 [nprobe.c:7940] Non IPv4/v6 traffic is discarded according to the template 17/Apr/2018 14:38:01 [util.c:440] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat 17/Apr/2018 14:38:01 [util.c:451] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat 17/Apr/2018 14:38:01 [nprobe.c:8622] IPv6 traffic will NOT be exported/accounted by this probe 17/Apr/2018 14:38:01 [nprobe.c:8623] due to configuration options (e.g. use NetFlow v9) 17/Apr/2018 14:38:01 [nprobe.c:8624] Please use -V to set the version to other than NetFlow V5 17/Apr/2018 14:38:01 [nprobe.c:8775] Not capturing packet from interface (collector mode) 17/Apr/2018 14:38:01 [util.c:4511] Initializing ZMQ as server 17/Apr/2018 14:38:01 [util.c:4554] Succesfully created ZMQ endpoint tcp://127.0.0.1:5556 17/Apr/2018 14:38:01 [util.c:3589] nProbe changed user to 'nobody' 17/Apr/2018 14:38:01 [collect.c:144] Flow collector listening on port 6343 (IPv4/v6) 17/Apr/2018 14:38:01 [nprobe.c:8992] nProbe started successfully

The ports are open and connected:

tcp        0      0 127.0.0.1:5556          0.0.0.0:*               LISTEN      7332/nprobe
tcp        0      0 127.0.0.1:5556          127.0.0.1:54708         ESTABLISHED 7332/nprobe
tcp        0      0 127.0.0.1:54708         127.0.0.1:5556          ESTABLISHED 15283/ntopng
udp        0      0 0.0.0.0:6343            0.0.0.0:*                           7332/nprobe
udp6       0      0 :::6343                 :::*                                7332/nprobe

Running tshark on the system, I can see traffic coming from my Cisco ASA to port 6343. I also have traffic on the loopback interface going between ntopng and nprobe:

tshark -i lo -c 10 -f 'port 5556'

Capturing on 'Loopback'
    1 0.000000000    127.0.0.1 → 127.0.0.1    TCP 322 5556 → 54708 [PSH, ACK] Seq=1 Ack=1 Win=342 Len=256 TSval=663468 TSecr=663218
    2 0.000014898    127.0.0.1 → 127.0.0.1    TCP 66 54708 → 5556 [ACK] Seq=1 Ack=257 Win=32480 Len=0 TSval=663468 TSecr=663468
    3 1.000429711    127.0.0.1 → 127.0.0.1    TCP 322 5556 → 54708 [PSH, ACK] Seq=257 Ack=1 Win=342 Len=256 TSval=663718 TSecr=663468
    4 1.000445166    127.0.0.1 → 127.0.0.1    TCP 66 54708 → 5556 [ACK] Seq=1 Ack=513 Win=32480 Len=0 TSval=663718 TSecr=663718
    5 2.000849371    127.0.0.1 → 127.0.0.1    TCP 323 5556 → 54708 [PSH, ACK] Seq=513 Ack=1 Win=342 Len=257 TSval=663968 TSecr=663718
    6 2.000863630    127.0.0.1 → 127.0.0.1    TCP 66 54708 → 5556 [ACK] Seq=1 Ack=770 Win=32480 Len=0 TSval=663968 TSecr=663968
    7 3.001326016    127.0.0.1 → 127.0.0.1    TCP 323 5556 → 54708 [PSH, ACK] Seq=770 Ack=1 Win=342 Len=257 TSval=664218 TSecr=663968
    8 3.001337520    127.0.0.1 → 127.0.0.1    TCP 66 54708 → 5556 [ACK] Seq=1 Ack=1027 Win=32480 Len=0 TSval=664218 TSecr=664218
    9 4.001763554    127.0.0.1 → 127.0.0.1    TCP 322 5556 → 54708 [PSH, ACK] Seq=1027 Ack=1 Win=342 Len=256 TSval=664468 TSecr=664218
   10 4.001781163    127.0.0.1 → 127.0.0.1    TCP 66 54708 → 5556 [ACK] Seq=1 Ack=1283 Win=32480 Len=0 TSval=664468 TSecr=664468

Despite everything looking fine, when I look at the ntopng web UI, I have nothing. No data available. Sometimes if I leave it running I'll get a short burst of data at some point, and then goes back to nothing. I had given up on it and left it running for several days and came back to notice it seeming running just fine which continued for a week or so until it quit again, and now I can't seem to get it going.

I didn't have any problems like these in the other setup, which was basically identical. Same equipment, same Ubuntu, same ntopng/nprobe configuration. One other odd thing I noticed is that there seem to be upgrades available for nprobe and pfring, but they won't upgrade without uninstalling ntopng because it requires the exact current version of pfring.

nprobe/now 8.2.180315-6003 amd64 [installed,upgradable to: 8.4.180417-6115]
ntopng/now 3.2.180315-4148 amd64 [installed,local]
ntopng-data/now 3.2.180315 all [installed,local]
pfring/now 7.0.0-1797 amd64 [installed,upgradable to: 7.0.0-1846]
pfring-dkms/unknown,now 7.0.0 all [installed,automatic]

Any thoughts?

Karl-MCS commented 6 years ago

Forgot one thing. When I ctrl+c ntopng I get this. It seems to indicate no packets have been received. And I don't know what's up with the pid file. I've tried removing it and rebooting the server, but it seems to happen again.

17/Apr/2018 15:12:38 [main.cpp:46] Shutting down...
17/Apr/2018 15:12:40 [ProtoStats.cpp:35] [IPv4]  0 B/0.00 Packets
17/Apr/2018 15:12:40 [ProtoStats.cpp:35] [IPv6]  0 B/0.00 Packets
17/Apr/2018 15:12:40 [ProtoStats.cpp:35] [ARP]   0 B/0.00 Packets
17/Apr/2018 15:12:40 [ProtoStats.cpp:35] [MPLS]  0 B/0.00 Packets
17/Apr/2018 15:12:40 [ProtoStats.cpp:35] [Other] 0 B/0.00 Packets
17/Apr/2018 15:12:41 [Ntop.cpp:1659] Interface tcp://127.0.0.1:5556 [running: 0]
17/Apr/2018 15:12:41 [main.cpp:65] Deleted PID /var/run/ntopng.pid: [rc: -1][Permission denied]
17/Apr/2018 15:12:41 [HTTPserver.cpp:929] HTTP server terminated
17/Apr/2018 15:12:41 [AddressResolution.cpp:61] Address resolution stats [0 resolved][0 failures]
emanuele-f commented 6 years ago

Hi, can you try to disable nprobe caching by adding the following options? --disable-cache --zmq-disable-buffering You may also want to move your options to config files and start nprobe and ntopng as services

Karl-MCS commented 6 years ago

I tried adding those two options to nprobe, but there's been no change. Looking at the interface on the webpage, it says...

Active Probe ZMQ Endpoints: 1 Collected Flows: 0 ZMQ Message Drops: 0 Received Traffic: 0 Bytes [0 Pkts]

It's like it doesn't see or process the flows, but I can see it receiving them on the loopback interface.

I do have nprobe and ntopng set up as services, but I have them stopped and running from CLI just for troubleshooting.

emanuele-f commented 6 years ago

Can you update ntopng and nprobe to the latest version and retry?

Karl-MCS commented 6 years ago

Just for kicks, I also put in --zmq-disable-compression and moved the settings back into the config files. I didn't get anything for a few minutes and just left it running. Came back and found it was working, but then a few minutes later stopped again and just shows zero flows.

I tried to update everything, but there seems to be a dependency issue. Apt shows there is a new version of nprobe and pfring, but not ntopng, and ntopng requires the current exact version of pfring Depends: pfring (= 7.0.0-1797). If I try to update it, it says it will uninstall ntopng. How do I fix that?

root@msms-ntop:/etc# apt list --installed | egrep '(nprobe|ntop|pfring)'
apt-ntop-stable/unknown,now 2.6-927 all [installed]
nprobe/now 8.2.180315-6003 amd64 [installed,upgradable to: 8.4.180417-6115]
ntopng/now 3.2.180315-4148 amd64 [installed,local]
ntopng-data/now 3.2.180315 all [installed,local]
pfring/now 7.0.0-1797 amd64 [installed,upgradable to: 7.0.0-1846]
pfring-dkms/unknown,now 7.0.0 all [installed,automatic]

root@msms-ntop:/etc# apt show ntopng
Package: ntopng
Version: 3.2.180315-4148
Status: install ok installed
Priority: optional
Section: free
Maintainer: Luca Deri <deri@ntop.org>
Installed-Size: 26.0 MB
Depends: pfring (= 7.0.0-1797), libsqlite3-0, libgeoip1, redis-server, librrd4, logrotate, libcurl3, libpcap0.8, libldap-2.4-2, libhiredis0.13, libssl1.0.0, libmysqlclient20, librdkafka1, lsb-release, ethtool, libcap2, bridge-utils, libnetfilter-conntrack3, udev, libnuma1, libzmq5, libnetfilter-queue1
Recommends: ntopng-data
Replaces: ntop
Download-Size: unknown
APT-Manual-Installed: yes
APT-Sources: /var/lib/dpkg/status
Description: Web-based traffic monitoring.

root@msms-ntop:/etc# apt full-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  bridge-utils fontconfig fontconfig-config fonts-dejavu-core libcairo2 libdatrie1 libdbi1 libfontconfig1 libgraphite2-3 libharfbuzz0b libpango-1.0-0
  libpangocairo-1.0-0 libpangoft2-1.0-0 libpixman-1-0 librrd4 libthai-data libthai0 libxcb-render0 libxcb-shm0 libxrender1
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  ntopng
The following packages will be upgraded:
  grub-legacy-ec2 nprobe pfring
3 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
Need to get 2,719 kB of archives.
After this operation, 22.2 MB disk space will be freed.
emanuele-f commented 6 years ago

With the apt-get command you need to:

Karl-MCS commented 6 years ago

Same thing, it says there isn't a new version of ntopng that works with the new version of pfring.

root@msms-ntop:/etc# apt update
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:5 http://packages.ntop.org/apt-stable/16.04 x64/ InRelease
Hit:6 http://packages.ntop.org/apt-stable/16.04 all/ InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.

root@msms-ntop:/etc# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  nprobe pfring
The following packages will be upgraded:
  grub-legacy-ec2
1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 24.3 kB of archives.
After this operation, 1,024 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 grub-legacy-ec2 all 18.2-4-g05926e48-0ubuntu1~16.04.1 [24.3 kB]
Fetched 24.3 kB in 0s (94.9 kB/s)
Preconfiguring packages ...
(Reading database ... 97978 files and directories currently installed.)
Preparing to unpack .../grub-legacy-ec2_18.2-4-g05926e48-0ubuntu1~16.04.1_all.deb ...
Leaving 'diversion of /usr/sbin/grub-set-default to /usr/sbin/grub-set-default.real by grub-legacy-ec2'
Unpacking grub-legacy-ec2 (18.2-4-g05926e48-0ubuntu1~16.04.1) over (17.2-35-gf576b2a2-0ubuntu1~16.04.2) ...
Setting up grub-legacy-ec2 (18.2-4-g05926e48-0ubuntu1~16.04.1) ...
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /vmlinuz-4.4.0-119-generic
Found kernel: /vmlinuz-4.4.0-116-generic
Found kernel: /vmlinuz-4.4.0-119-generic
Found kernel: /vmlinuz-4.4.0-116-generic
Updating /boot/grub/menu.lst ... done

root@msms-ntop:/etc# apt install ntopng nprobe
Reading package lists... Done
Building dependency tree
Reading state information... Done
ntopng is already the newest version (3.2.180315-4148).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 nprobe : Depends: pfring (= 7.0.0-1846) but 7.0.0-1797 is to be installed
E: Unable to correct problems, you have held broken packages.
emanuele-f commented 6 years ago

What's the output of apt-cache policy pfring ? From your output above it seems like new pfring version is recognized, have you pinned the package?

pfring/now 7.0.0-1797 amd64 [installed,upgradable to: 7.0.0-1846]
Karl-MCS commented 6 years ago

There shouldn't be any pinning or non-default package holds. This is a virtual server that was installed just to run ntopng, so it's a clean install and has never been used for anything else. It's my 2nd setup like this and I didn't have any of these problems on the other one. Very strange...

Apt-cache seems to be aware of ntopng version 3.2.2.180417-4328 but for whatever reason is not selecting it. The folder /etc/apt/preferences.d is empty.

root@msms-ntop:/etc# apt-cache policy pfring
pfring:
  Installed: 7.0.0-1797
  Candidate: 7.0.0-1846
  Version table:
     7.0.0-1846 500
        500 http://packages.ntop.org/apt-stable/16.04 x64/ Packages
 *** 7.0.0-1797 100
        100 /var/lib/dpkg/status
root@msms-ntop:/etc# apt-cache policy nprobe
nprobe:
  Installed: 8.2.180315-6003
  Candidate: 8.4.180417-6115
  Version table:
     8.4.180417-6115 500
        500 http://packages.ntop.org/apt-stable/16.04 x64/ Packages
 *** 8.2.180315-6003 100
        100 /var/lib/dpkg/status
root@msms-ntop:/etc# apt-cache policy ntopng
ntopng:
  Installed: 3.2.180315-4148
  Candidate: 3.2.180315-4148
  Version table:
 *** 3.2.180315-4148 100
        100 /var/lib/dpkg/status
     3.2.2.180417-4328 500
        500 http://packages.ntop.org/apt-stable/16.04 x64/ Packages
     2.2+dfsg1-1build1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages

root@msms-ntop:/etc/apt# dpkg --get-selections | egrep '(ntop|nprobe|pfring)'
apt-ntop-stable                                 install
nprobe                                          install
ntopng                                          install
ntopng-data                                     install
pfring                                          install
pfring-dkms                                     install
Karl-MCS commented 6 years ago

Okay... I don't know what was going on there, but I just did apt remove ntopng and apt install ntopng and it updated everything.

It still says no data available.

mzac commented 6 years ago

Similar issue here, can't upgrade ntopng on ubuntu 16.04, do I remove it?

root@ntop1:~# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  cento n2disk nprobe pfring
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.

root@ntop1:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  bridge-utils
Use 'apt autoremove' to remove it.
The following packages will be REMOVED:
  ntopng
The following packages will be upgraded:
  cento n2disk nprobe pfring
4 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
Need to get 4,328 kB of archives.
After this operation, 26.1 MB disk space will be freed.
Do you want to continue? [Y/n]
Karl-MCS commented 6 years ago

@mzac That seemed to fix the package issue for me. After removing, when I reinstalled it went to the later version and also updated nprobe and pfring. Ntopng and nprobe depend on a specific version of pfring, so if any of them don't update, they all get held. I don't know why it wouldn't do an in-place upgrade.

Unfortunately this didn't resolve my no data issue.

Karl-MCS commented 6 years ago

I just threw together a TurnKey Core 14.2 virtual machine and installed ntopng and nprobe using the same config, and added it as a flow destination in my Cisco ASA, and it's working just fine. So I don't know what's going on with this Ubuntu VM (the other working one was also Ubuntu), but my config seems to be working fine.

Karl-MCS commented 6 years ago

Okay, so as a last resort, I purged all of the ntop-related packages, cleared the apt cache, redownloaded all of it, rewrote the configs, and started over. It still doesn't work. I believe the configuration is good because it's working on the other VM running in demo mode, and a basically identical setup is working fine at another site.

Is there anything else I can try before I try to get this license moved?

mzac commented 6 years ago

@Karl-MCS You say your setup is a VM, did you setup the NIC in your hypervisor to use promiscuous mode? Also you may need to tag it with all vlans (4095)

https://blog.packet-foo.com/2013/04/capturing-packets-of-vmware-machines/ https://kb.vmware.com/s/article/1038847 http://vmnomad.blogspot.ca/2011/07/vlan-tagging-and-use-cases-of-vlan-id.html

Karl-MCS commented 6 years ago

I'm only collecting via NetFlow, so the interface settings should be fine. There is a problem somewhere with either nProbe or ntopng. I'm leaning toward ntopng simply because I can see nProbe handing off zmq to it on the loopback interface, but ntopng reports no flows received.

I still have the TurnKey VM I set up yesterday which was working, so what I may do is use the nProbe on my Ubuntu VM and have it send zmq to ntopng on the TurnKey VM, or vice versa. Maybe then I can tell which one is having the problem.

emanuele-f commented 6 years ago

You can use the -b2 flag in nprobe and the -v3 flag in ntopng to print on the console the exported flows. This way you chould figure out if the problem is on the ntopng or on the nprobe side

Karl-MCS commented 6 years ago

Okay, I tried this this morning, and here is what I see. Based on this I think the problem is likely with nProbe because even with level 2 verbose logging, I don't see any additional output at all from either nprobe or ntopng once they start. I do see nprobe receiving flows from the firewall, and I also see ntopng polling nprobe for flows, but I suspect nprobe is not providing them.

sudo nprobe -i none -n none --zmq "tcp://127.0.0.1:5556" --collector-port 6343 --online-license-check --verbose 2

23/Apr/2018 11:00:41 [plugin.c:175] No plugins found in ./plugins
23/Apr/2018 11:00:41 [plugin.c:183] Loading 23 plugins [.so] from /usr/local/lib/nprobe/plugins
23/Apr/2018 11:00:41 [nprobe.c:2272] Contacting licensing server. Please hold on...
23/Apr/2018 11:00:41 [nprobe.c:3780] Valid nProbe Pro license found
23/Apr/2018 11:00:41 [nprobe.c:5388] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
23/Apr/2018 11:00:41 [nprobe.c:5391] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
23/Apr/2018 11:00:41 [nprobe.c:5478] Welcome to nProbe Pro v.8.4.180417 ($Revision: 6115 $) for x86_64-pc-linux-gnu with native PF_RING acceleration
23/Apr/2018 11:00:41 [nprobe.c:5488] Running on Ubuntu 16.04.4 LTS
23/Apr/2018 11:00:41 [nprobe.c:5499] [LICENSE] nProbe SystemId: ****************
23/Apr/2018 11:00:41 [nprobe.c:5503] Tracing enabled
23/Apr/2018 11:00:41 [nprobe.c:5566] Sample rate [packet: 1][flow collection/export: 1/1]
23/Apr/2018 11:00:41 [bgpPlugin.c:404] BGP plugin is disabled (--bgp-port has not been specified)
23/Apr/2018 11:00:41 [dbPlugin.c:48] Initializing DB plugin
23/Apr/2018 11:00:41 [dhcpPlugin.c:305] Initialized DHCP plugin
23/Apr/2018 11:00:41 [diameterPlugin.c:101] Initialized Diameter plugin
23/Apr/2018 11:00:41 [dnsPlugin.c:61] Initialized DNS plugin
23/Apr/2018 11:00:41 [exportPlugin.c:356] Initializing Export plugin
23/Apr/2018 11:00:41 [ftpPlugin.c:80] Initialized FTP plugin
23/Apr/2018 11:00:41 [gtpv0Plugin.c:92] Initialized GTPv0 plugin
23/Apr/2018 11:00:41 [gtpv1Plugin.c:122] Initialized GTPv1 plugin
23/Apr/2018 11:00:41 [gtpv2Plugin.c:253] Initialized GTPv2 plugin
23/Apr/2018 11:00:41 [httpPlugin.c:250] HTTP log files will be dumped each 60 seconds or each 10000 lines
23/Apr/2018 11:00:41 [httpPlugin.c:257] Initialized HTTP plugin
23/Apr/2018 11:00:41 [imapPlugin.c:127] Initialized IMAP plugin
23/Apr/2018 11:00:41 [mysqlPlugin.c:111] Initialized MySQL plugin
23/Apr/2018 11:00:41 [netbiosPlugin.c:50] Initialized NETBIOS plugin
23/Apr/2018 11:00:41 [nflitePlugin.c:914] [NFLite] Initialized NetFlow-Lite plugin
23/Apr/2018 11:00:41 [oraclePlugin.c:173] Initialized Oracle plugin
23/Apr/2018 11:00:41 [popPlugin.c:118] Initialized POP plugin
23/Apr/2018 11:00:41 [radiusPlugin.c:124] Initialized Radius plugin
23/Apr/2018 11:00:41 [rtpPlugin.c:173] Initializing RTP plugin [argc: 12]
23/Apr/2018 11:00:41 [s1apPlugin.c:1407] Initialized S1AP plugin
23/Apr/2018 11:00:41 [sipPlugin.c:261] Initialized SIP plugin
23/Apr/2018 11:00:41 [sipPlugin.c:288] Initialized SIP plugin
23/Apr/2018 11:00:41 [smtpPlugin.c:119] Initialized SMTP plugin
23/Apr/2018 11:00:41 [ssdpPlugin.c:54] Initialized SSDP plugin
23/Apr/2018 11:00:41 [plugin.c:259] 23 plugin(s) loaded [21 delete][20 packet].
23/Apr/2018 11:00:41 [nprobe.c:8250] Welcome to nProbe v.8.4.180417 for x86_64-pc-linux-gnu
23/Apr/2018 11:00:41 [nprobe.c:7218] Compiling flow templates...
23/Apr/2018 11:00:41 [nprobe.c:7160] WARNING: Adding %EXPORTER_IPV4_ADDRESS to the template as nProbe is working as collector
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin BGP Update Listener [bgp]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin MySQL DB [db]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin DHCP Protocol [dhcp]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin Diameter Protocol [diameter]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin DNS/LLMNR Protocol [dns]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin Export Plugin [export]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin FTP Protocol [ftp]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin GTPv0 Signaling Protocol [gtpv0]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin GTPv1 Signaling Protocol [gtpv1]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin GTPv2 Signaling Protocol [gtpv2]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin HTTP Protocol [http]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin IMAP Protocol [imap]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin MySQL Plugin [mysql]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin NETBIOS Protocol [netbios]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin Netflow-Lite Plugin [nflite]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin Oracle Protocol [oracle]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin POP3 Protocol [pop3]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin Radius Protocol [radius]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin RTP Plugin [rtp]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin S1AP Protocol [S1AP]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin SIP Plugin [sip]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin SMTP Protocol [smtp]
23/Apr/2018 11:00:41 [plugin.c:960] Scanning plugin SSDP Protocol [ssdp]
23/Apr/2018 11:00:41 [plugin.c:1212] 0 plugin(s) enabled
23/Apr/2018 11:00:41 [nprobe.c:7730] Non IPv4/v6 traffic is discarded according to the template
23/Apr/2018 11:00:41 [util.c:451] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
23/Apr/2018 11:00:41 [util.c:462] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
23/Apr/2018 11:00:41 [nprobe.c:8427] IPv6 traffic will NOT be exported/accounted by this probe
23/Apr/2018 11:00:41 [nprobe.c:8428] due to configuration options (e.g. use NetFlow v9)
23/Apr/2018 11:00:41 [nprobe.c:8429] Please use -V to set the version to other than NetFlow V5
23/Apr/2018 11:00:41 [nprobe.c:8432] The flows hash has 131072 buckets
23/Apr/2018 11:00:41 [nprobe.c:8434] Flows older than 120 seconds will be exported
23/Apr/2018 11:00:41 [nprobe.c:8437] Flows inactive for at least 30 seconds will be exported
23/Apr/2018 11:00:41 [nprobe.c:8440] Expired flows will not be queued for more than 30 seconds
23/Apr/2018 11:00:41 [nprobe.c:8447] Exported flows with engineType 0 and engineId 41
23/Apr/2018 11:00:41 [nprobe.c:8507] Flows will be emitted in NetFlow 5 format
23/Apr/2018 11:00:41 [nprobe.c:8560] Flow input interface index is set to 0
23/Apr/2018 11:00:41 [nprobe.c:8566] Flow output interface index is set to 0
23/Apr/2018 11:00:41 [nprobe.c:8580] Not capturing packet from interface (collector mode)
23/Apr/2018 11:00:41 [util.c:4529] Initializing ZMQ as server
23/Apr/2018 11:00:41 [util.c:4572] Succesfully created ZMQ endpoint tcp://127.0.0.1:5556
23/Apr/2018 11:00:41 [util.c:3607] nProbe changed user to 'nobody'
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin BGP Update Listener (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin MySQL DB (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin DHCP Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin Diameter Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin DNS/LLMNR Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin Export Plugin (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin FTP Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin GTPv0 Signaling Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin GTPv1 Signaling Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin GTPv2 Signaling Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin HTTP Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin IMAP Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin MySQL Plugin (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin NETBIOS Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin Netflow-Lite Plugin (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin Oracle Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin POP3 Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin Radius Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin RTP Plugin (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin S1AP Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin SIP Plugin (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin SMTP Protocol (no template is using it)
23/Apr/2018 11:00:41 [plugin.c:918] Disabling plugin SSDP Protocol (no template is using it)
23/Apr/2018 11:00:41 [collect.c:85] Created UDP sockets
23/Apr/2018 11:00:41 [collect.c:89] Created a SCTP socket (17)
23/Apr/2018 11:00:41 [collect.c:144] Flow collector listening on port 6343 (IPv4/v6)
23/Apr/2018 11:00:41 [nprobe.c:8703] Starting 1 packet fetch thread(s)
23/Apr/2018 11:00:41 [engine.c:3595] Starting bucket dequeue thread
23/Apr/2018 11:00:41 [nprobe.c:8811] nProbe started successfully
sudo ntopng --interface "tcp://127.0.0.1:5556" --http-port 80 --local-networks "192.168.0.0/16,172.16.0.0/12,10.0.0.0/8" --online-license-check --verbose 2

23/Apr/2018 10:59:48 [Ntop.cpp:1499] Setting local networks to 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8
23/Apr/2018 10:59:48 [Redis.cpp:127] Successfully connected to redis 127.0.0.1:6379@0
23/Apr/2018 10:59:48 [Redis.cpp:127] Successfully connected to redis 127.0.0.1:6379@0
23/Apr/2018 10:59:48 [NtopPro.cpp:262] [LICENSE] Reading license from /etc/ntopng.license
23/Apr/2018 10:59:48 [NtopPro.cpp:157] Contacting licensing server. Please hold on...
23/Apr/2018 10:59:49 [NtopPro.cpp:175] [LICENSE] /etc/ntopng.license: found valid Enterprise license
23/Apr/2018 10:59:49 [Ntop.cpp:1626] Registered interface tcp://127.0.0.1:5556 [id: 2]
23/Apr/2018 10:59:49 [main.cpp:304] PID stored in file /var/run/ntopng.pid
23/Apr/2018 10:59:50 [HTTPserver.cpp:860] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL.
23/Apr/2018 10:59:50 [Utils.cpp:469] User changed to nobody
23/Apr/2018 10:59:50 [HTTPserver.cpp:931] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
23/Apr/2018 10:59:50 [HTTPserver.cpp:934] HTTP server listening on port(s) 80
23/Apr/2018 10:59:50 [main.cpp:386] Working directory: /var/tmp/ntopng
23/Apr/2018 10:59:50 [main.cpp:388] Scripts/HTML pages directory: /usr/share/ntopng
23/Apr/2018 10:59:50 [Ntop.cpp:391] Welcome to ntopng x86_64 v.3.2.2.180417 - (C) 1998-18 ntop.org
23/Apr/2018 10:59:50 [Ntop.cpp:401] Built on Ubuntu 16.04.4 LTS
23/Apr/2018 10:59:50 [NtopPro.cpp:508] [LICENSE] System Id:     688D729C5505617E
23/Apr/2018 10:59:50 [NtopPro.cpp:509] [LICENSE] Edition:       Enterprise
23/Apr/2018 10:59:50 [NtopPro.cpp:510] [LICENSE] License Type:  Permanent License
23/Apr/2018 10:59:50 [NtopPro.cpp:513] [LICENSE] License:       52096FDD431144ED0DB588D2EC4C91CE15513749602485141B
23/Apr/2018 10:59:50 [NtopPro.cpp:523] [LICENSE] Maintenance:   Until Thu Feb 28 11:29:20 2019 [311 days left]
23/Apr/2018 10:59:50 [PeriodicActivities.cpp:59] Started periodic activities loop...
23/Apr/2018 10:59:50 [PeriodicActivities.cpp:100] Each periodic activity script will use 2 threads
23/Apr/2018 10:59:50 [NetworkInterface.cpp:2472] Started packet polling on interface tcp://127.0.0.1:5556 [id: 2]...
23/Apr/2018 10:59:50 [CollectorInterface.cpp:122] Collecting flows on tcp://127.0.0.1:5556
tshark -i any -f 'port 6343 or port 5556' -c 20

Capturing on 'any'
    1 0.000000000    firewall-ip-address → nprobe-ip-address UDP 1456 25305 → 6343 Len=1412
    2 0.160567423    firewall-ip-address → nprobe-ip-address UDP 1496 25305 → 6343 Len=1452
    3 0.418897825    127.0.0.1 → 127.0.0.1    TCP 325 5556 → 46600 [PSH, ACK] Seq=1 Ack=1 Win=342 Len=257 TSval=107720707 TSecr=107720457
    4 0.418913668    127.0.0.1 → 127.0.0.1    TCP 68 46600 → 5556 [ACK] Seq=1 Ack=258 Win=32480 Len=0 TSval=107720707 TSecr=107720707
    5 0.557819482    firewall-ip-address → nprobe-ip-address UDP 1464 25305 → 6343 Len=1420
    6 0.655837821    firewall-ip-address → nprobe-ip-address UDP 1476 25305 → 6343 Len=1432
    7 1.038357116    firewall-ip-address → nprobe-ip-address UDP 1444 25305 → 6343 Len=1400
    8 1.193978912    firewall-ip-address → nprobe-ip-address UDP 1504 25305 → 6343 Len=1460
    9 1.316132229    firewall-ip-address → nprobe-ip-address UDP 1480 25305 → 6343 Len=1436
   10 1.419251294    127.0.0.1 → 127.0.0.1    TCP 325 5556 → 46600 [PSH, ACK] Seq=258 Ack=1 Win=342 Len=257 TSval=107720957 TSecr=107720707
   11 1.419263444    127.0.0.1 → 127.0.0.1    TCP 68 46600 → 5556 [ACK] Seq=1 Ack=515 Win=32480 Len=0 TSval=107720957 TSecr=107720957
   12 1.548025399    firewall-ip-address → nprobe-ip-address UDP 1456 25305 → 6343 Len=1412
   13 1.795826804    firewall-ip-address → nprobe-ip-address UDP 1476 25305 → 6343 Len=1432
   14 2.024262204    firewall-ip-address → nprobe-ip-address UDP 1516 25305 → 6343 Len=1472
   15 2.251613188    firewall-ip-address → nprobe-ip-address UDP 1516 25305 → 6343 Len=1472
   16 2.291291493    firewall-ip-address → nprobe-ip-address UDP 1460 25305 → 6343 Len=1416
   17 2.419936551    127.0.0.1 → 127.0.0.1    TCP 324 5556 → 46600 [PSH, ACK] Seq=515 Ack=1 Win=342 Len=256 TSval=107721208 TSecr=107720957
   18 2.419952025    127.0.0.1 → 127.0.0.1    TCP 68 46600 → 5556 [ACK] Seq=1 Ack=771 Win=32480 Len=0 TSval=107721208 TSecr=107721208
   19 2.466874456    firewall-ip-address → nprobe-ip-address UDP 1456 25305 → 6343 Len=1412
   20 2.715543907    firewall-ip-address → nprobe-ip-address UDP 1504 25305 → 6343 Len=1460
Karl-MCS commented 6 years ago

I see nprobe and ntopng were updated over the weekend, so as I was shutting down to update, nprobe said this. What does this mean when it collected packets but did not process any flows?

23/Apr/2018 11:14:45 [nprobe.c:5602] nProbe is shutting down...
23/Apr/2018 11:14:45 [nprobe.c:5626] Exporting pending buckets...
23/Apr/2018 11:14:45 [nprobe.c:5655] Pending buckets have been exported...
23/Apr/2018 11:14:47 [engine.c:3678] Export thread terminated [exportQueue=0]
23/Apr/2018 11:14:47 [nprobe.c:5731] Flushing queued flows...
23/Apr/2018 11:14:47 [nprobe.c:5739] Freeing memory...
23/Apr/2018 11:14:47 [plugin.c:288] Terminating plugins.
23/Apr/2018 11:14:47 [nprobe.c:5864] Still allocated 0 hash buckets
23/Apr/2018 11:14:47 [nprobe.c:3030] Processed packets: 0 (max bucket search: 0)
23/Apr/2018 11:14:47 [nprobe.c:3013] Fragment queue length: 0
23/Apr/2018 11:14:47 [nprobe.c:3039] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
23/Apr/2018 11:14:47 [nprobe.c:3046] Flow collection:   [collected pkts: 1089][processed flows: 0]
23/Apr/2018 11:14:47 [nprobe.c:3049] Flow drop stats:   [0 bytes/0 pkts][0 flows]
23/Apr/2018 11:14:47 [nprobe.c:3054] Total flow stats:  [0 bytes/0 pkts][0 flows/0 pkts sent]
23/Apr/2018 11:14:47 [nprobe.c:5873] Cleaning globals
23/Apr/2018 11:14:47 [nprobe.c:5892] nProbe terminated.
Karl-MCS commented 6 years ago

I installed the below updates. After it started back up, I got a brief burst of data on the webpage, and then it stopped again.

linux-generic/xenial-updates,xenial-security 4.4.0.121.127 amd64 [upgradable from: 4.4.0.119.125]
linux-headers-generic/xenial-updates,xenial-security 4.4.0.121.127 amd64 [upgradable from: 4.4.0.119.125]
linux-image-generic/xenial-updates,xenial-security 4.4.0.121.127 amd64 [upgradable from: 4.4.0.119.125]
linux-libc-dev/xenial-updates,xenial-security 4.4.0-121.145 amd64 [upgradable from: 4.4.0-119.143]
nprobe/unknown 8.4.180420-6115 amd64 [upgradable from: 8.4.180417-6115]
ntopng/unknown 3.4.180420-4342 amd64 [upgradable from: 3.2.2.180417-4328]
ntopng-data/unknown 3.4.180420 all [upgradable from: 3.2.2.180417]
pfring/unknown 7.0.0-1859 amd64 [upgradable from: 7.0.0-1846]
emanuele-f commented 6 years ago

If you run nprobe with -b2 option and you get no flow export messages, then the problem is that the netflow traffic can't reach the nprobe process. Please note that even if you can see the packets with tcpdump/tshark they may be dropped by a local firewall rule.

Please run the following script to verify that the nprobe host can actually receive the netflow data:

#!/usr/bin/python
import socket
port = 6343

UDPSock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)

listen_addr = ("",port)
UDPSock.bind(listen_addr)

print "Listening on port "+str(port)

while True:
        data,addr = UDPSock.recvfrom(1024)
        print "Received packet from ",addr
Karl-MCS commented 6 years ago

There is no local firewall active on the system.

Listening on port 6343
Received packet from  ('xxx.xxx.xxx.xxx', 25305)
Received packet from  ('xxx.xxx.xxx.xxx', 25305)
Received packet from  ('xxx.xxx.xxx.xxx', 25305)
Received packet from  ('xxx.xxx.xxx.xxx', 25305)
Received packet from  ('xxx.xxx.xxx.xxx', 25305)
Received packet from  ('xxx.xxx.xxx.xxx', 25305)

It seems to me the below output while terminating nprobe shows it has received the packets but taken no action on them.

23/Apr/2018 11:14:47 [nprobe.c:5864] Still allocated 0 hash buckets
23/Apr/2018 11:14:47 [nprobe.c:3030] Processed packets: 0 (max bucket search: 0)
23/Apr/2018 11:14:47 [nprobe.c:3013] Fragment queue length: 0
23/Apr/2018 11:14:47 [nprobe.c:3039] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
23/Apr/2018 11:14:47 [nprobe.c:3046] Flow collection:   [collected pkts: 1089][processed flows: 0]
23/Apr/2018 11:14:47 [nprobe.c:3049] Flow drop stats:   [0 bytes/0 pkts][0 flows]
23/Apr/2018 11:14:47 [nprobe.c:3054] Total flow stats:  [0 bytes/0 pkts][0 flows/0 pkts sent]
emanuele-f commented 6 years ago

Can you provide a pcap of the netflow traffic as seen as the nprobe host?

emanuele-f commented 6 years ago

Closing the issue as the conversation continued via email.

nygc-jmaldonado commented 3 years ago

What was the outcome of this? I am experiencing the same issue