ntop / nProbe

Open source components and extensions for nProbe
http://ntop.org
GNU General Public License v2.0
1.65k stars 44 forks source link

nProbe does not connect to mysql after update #301

Closed romeor closed 5 years ago

romeor commented 5 years ago

Hello,

As requested, I open an issue here. Recently I updated nprobe running apt-get update and apt-get dist-upgrade for debian 9.1 repo. I was running the stable version, don't mind the exact version number, but it was just the one latest before the update in october I guess, as we started to test ntop in septempber and everything worked ok. I run nprobe with mysql for historical data as netflow collector.

After the update was done, I restarted the VM running nprobe and ntopng, as there was the kernel and some module update. After restart there was no historical data available. I see the traffic graphs history, I see the recent netflow data on the graphs, but as soon as I switch to flows, talkers, protocols - there is no data to see. It just says - "Flow Search Results No results found. Please modify your search criteria."

After some debugging, I've noticed, that nprobe does not even try conenct to mysql to create new database running this command:

nprobe --zmq tcp://127.0.0.1:5556 --zmq-probe-mode --interface none --collector-port 2055 --collector none --flow-templ "@NTOPNG@%EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS" flow-version 9 --mysql 127.0.0.1:ntop:nf:roman:PASSWORD

But if I run

nprobe --zmq tcp://127.0.0.1:5556 --zmq-probe-mode --interface ens18 --flow-templ "@NTOPNG@%EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS" flow-version 9 --mysql 127.0.0.1:ntop:nf:roman:PASSWORD

I see new DB created inside the MariaDB console interface. So it seems, like nProbe does not connect to mysql in collector mode after latest changes.

as requested by e-mail, I've added the -b=2 --debug and here is the output:

# nprobe --zmq tcp://127.0.0.1:5556 --zmq-probe-mode --interface none -3 2055 --collector none --flow-templ "@NTOPNG@%EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS" flow-version 9 --mysql 127.0.0.1:ntop:nf:roman:PASSWORD -b=2 --debug

23/Oct/2018 10:01:29 [plugin.c:179] No plugins found in ./plugins 23/Oct/2018 10:01:29 [plugin.c:187] Loading 25 plugins [.so] from /usr/local/lib/nprobe/plugins 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin DHCP Protocol [/etc/nprobe.license.dhcp]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin Diameter Protocol [/etc/nprobe.license.diameter]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin DICOM Protocol [/etc/nprobe.license.dicom]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin DNS/LLMNR Protocol [/etc/nprobe.license.dns]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin Export Plugin [/etc/nprobe.license.export]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin FTP Protocol [/etc/nprobe.license.ftp]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin GTPv0 Signaling Protocol [/etc/nprobe.license.gtpv0]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin GTPv1 Signaling Protocol [/etc/nprobe.license.gtpv1]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin GTPv2 Signaling Protocol [/etc/nprobe.license.gtpv2]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin HTTP Protocol [/etc/nprobe.license.http]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin IMAP Protocol [/etc/nprobe.license.email]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin Netflow-Lite Plugin [/etc/nprobe.license.nflite]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin Oracle Protocol [/etc/nprobe.license.oracle]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin POP3 Protocol [/etc/nprobe.license.email]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin Radius Protocol [/etc/nprobe.license.radius]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin RTP Plugin [/etc/nprobe.license.voip]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin S1AP Protocol [/etc/nprobe.license.S1AP]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin SIP Plugin [/etc/nprobe.license.voip]: Missing license file 23/Oct/2018 10:01:29 [plugin.c:855] Unable to enable plugin SMTP Protocol [/etc/nprobe.license.email]: Missing license file 23/Oct/2018 10:01:29 [nprobe.c:4165] Valid nProbe Pro license found 23/Oct/2018 10:01:29 [nprobe.c:6082] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 23/Oct/2018 10:01:29 [nprobe.c:6085] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 23/Oct/2018 10:01:29 [nprobe.c:6172] Welcome to nProbe Pro v.8.7.181022 ($Revision: 6315 $) for x86_64-pc-linux-gnu with native PF_RING acceleration 23/Oct/2018 10:01:29 [nprobe.c:6182] Running on Debian GNU/Linux 9.1 (stretch) 23/Oct/2018 10:01:29 [nprobe.c:6193] [LICENSE] nProbe SystemId: B02E5E9A5904A1D2 23/Oct/2018 10:01:29 [nprobe.c:6260] Sample rate [packet: 1][flow collection/export: 1/1] 23/Oct/2018 10:01:29 [nprobe.c:8939] Welcome to nProbe v.8.7.181022 for x86_64-pc-linux-gnu 23/Oct/2018 10:01:29 [cache.c:1230] init_lru_cache(max_size=16384) 23/Oct/2018 10:01:29 [cache.c:1230] init_lru_cache(max_size=16384) 23/Oct/2018 10:01:29 [nprobe.c:7949] Using NetFlow Packet Payload Len: 1472 23/Oct/2018 10:01:29 [nprobe.c:7879] @NTOPNG@ expanded to " %L7_PROTO %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %SRC_VLAN %EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS" 23/Oct/2018 10:01:29 [template.c:2293] Processing %L7_PROTO %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %SRC_VLAN %EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS 23/Oct/2018 10:01:29 [template.c:2396] Checking [L7_PROTO][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV4_SRC_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV4_DST_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_SRC_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_DST_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV6_SRC_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV6_DST_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IP_PROTOCOL_VERSION][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [PROTOCOL][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [FIRST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [LAST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [SRC_VLAN][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [EXPORTER_IPV4_ADDRESS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [EXPORTER_IPV6_ADDRESS][found=1] 23/Oct/2018 10:01:29 [template.c:2293] Processing %L7_PROTO %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV4_SRC_ADDR %IPV4_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %SRC_VLAN %EXPORTER_IPV4_ADDRESS %EXPORTER_IPV4_ADDRESS 23/Oct/2018 10:01:29 [template.c:2396] Checking [L7_PROTO][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV4_SRC_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV4_DST_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_SRC_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_DST_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IP_PROTOCOL_VERSION][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [PROTOCOL][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [FIRST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [LAST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [SRC_VLAN][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [EXPORTER_IPV4_ADDRESS][found=1] 23/Oct/2018 10:01:29 [template.c:2293] Processing %L7_PROTO %IPV6_SRC_ADDR %IPV6_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %SRC_VLAN %EXPORTER_IPV6_ADDRESS %EXPORTER_IPV6_ADDRESS 23/Oct/2018 10:01:29 [template.c:2396] Checking [L7_PROTO][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV6_SRC_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV6_DST_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_SRC_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_DST_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IP_PROTOCOL_VERSION][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [PROTOCOL][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [FIRST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [LAST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [SRC_VLAN][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [EXPORTER_IPV6_ADDRESS][found=1] 23/Oct/2018 10:01:29 [template.c:2293] Processing %L7_PROTO %IPV6_SRC_ADDR %IPV6_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %SRC_VLAN %EXPORTER_IPV6_ADDRESS %EXPORTER_IPV6_ADDRESS 23/Oct/2018 10:01:29 [template.c:2396] Checking [L7_PROTO][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV6_SRC_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV6_DST_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_SRC_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_DST_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IP_PROTOCOL_VERSION][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [PROTOCOL][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [FIRST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [LAST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [SRC_VLAN][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [EXPORTER_IPV6_ADDRESS][found=1] 23/Oct/2018 10:01:29 [plugin.c:1272] 0 plugin(s) enabled 23/Oct/2018 10:01:29 [template.c:2293] Processing %IN_BYTES %IN_PKTS %PROTOCOL %L4_SRC_PORT %IPV4_SRC_ADDR %L4_DST_PORT %IPV4_DST_ADDR %LAST_SWITCHED %FIRST_SWITCHED %OUT_BYTES %OUT_PKTS %SRC_VLAN %IP_PROTOCOL_VERSION %EXPORTER_IPV4_ADDRESS %L7_PROTO 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [PROTOCOL][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_SRC_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV4_SRC_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_DST_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV4_DST_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [LAST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [FIRST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [SRC_VLAN][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IP_PROTOCOL_VERSION][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [EXPORTER_IPV4_ADDRESS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L7_PROTO][found=1] 23/Oct/2018 10:01:29 [template.c:2293] Processing %IN_BYTES %IN_PKTS %PROTOCOL %L4_SRC_PORT %L4_DST_PORT %LAST_SWITCHED %FIRST_SWITCHED %OUT_BYTES %OUT_PKTS %IPV6_SRC_ADDR %IPV6_DST_ADDR %SRC_VLAN %IP_PROTOCOL_VERSION %EXPORTER_IPV6_ADDRESS %L7_PROTO 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IN_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [PROTOCOL][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_SRC_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L4_DST_PORT][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [LAST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [FIRST_SWITCHED][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_BYTES][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [OUT_PKTS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV6_SRC_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IPV6_DST_ADDR][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [SRC_VLAN][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [IP_PROTOCOL_VERSION][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [EXPORTER_IPV6_ADDRESS][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [L7_PROTO][found=1] 23/Oct/2018 10:01:29 [template.c:2293] Processing %TOTAL_FLOWS_EXP %TOTAL_PKTS_EXP 23/Oct/2018 10:01:29 [template.c:2396] Checking [TOTAL_FLOWS_EXP][found=1] 23/Oct/2018 10:01:29 [template.c:2396] Checking [TOTAL_PKTS_EXP][found=1] 23/Oct/2018 10:01:29 [nprobe.c:8395] Each flow is 82 bytes long 23/Oct/2018 10:01:29 [nprobe.c:8396] The # flows per packet has been set to 16 23/Oct/2018 10:01:29 [nprobe.c:8399] IP TOS is accounted 23/Oct/2018 10:01:29 [nprobe.c:8425] Non IPv4/v6 traffic is discarded according to the template 23/Oct/2018 10:01:29 [util.c:507] Loaded database /usr/share/ntopng/httpdocs/geoip/GeoLite2-ASN.mmdb [ip_version: 6] 23/Oct/2018 10:01:29 [nprobe.c:9307] Not capturing packet from interface (collector mode) 23/Oct/2018 10:01:29 [util.c:4719] Initializing ZMQ as client 23/Oct/2018 10:01:29 [util.c:4738] Exporting flows towards ZMQ endpoint tcp://127.0.0.1:5556 23/Oct/2018 10:01:29 [util.c:3788] nProbe changed user to 'nprobe' 23/Oct/2018 10:01:29 [collect.c:142] Flow collector listening on port 2055 (IPv4/v6) 23/Oct/2018 10:01:29 [nprobe.c:9405] WARNING: 23/Oct/2018 10:01:29 [nprobe.c:9406] WARNING: You're running nprobe in DEBUG mode 23/Oct/2018 10:01:29 [nprobe.c:9407] WARNING: 23/Oct/2018 10:01:29 [nprobe.c:9555] nProbe started successfully 23/Oct/2018 10:01:30 [collect.c:2223] NETFLOW_DEBUG: Received 324 bytes flow 23/Oct/2018 10:01:30 [collect.c:854] +++ Dissecting packet 1 23/Oct/2018 10:01:30 [util.c:4802] [ZMQ] [event] { "iface": { "name": "none", "speed": 1000, "ip": "" }, "probe": { "ip": "172.21.0.200", "public_ip": "" }, "time" : 1540278090.523, "bytes": 0, "packets": 0, "avg": { "bps": 0, "pps": 0 }, "drops" : { "export_queue_too_long": 0, "too_many_flows": 0, "elk_flow_drops": 0, "sflow_pkt_sample_drops": 0 }, "timeout": { "lifetime": 120, "idle": 30 }, "zmq": { "num_flow_exports": 0, "num_exporters": 1 }, } 23/Oct/2018 10:01:31 [collect.c:2223] NETFLOW_DEBUG: Received 264 bytes flow 23/Oct/2018 10:01:31 [collect.c:854] +++ Dissecting packet 2 23/Oct/2018 10:01:31 [collect.c:2223] NETFLOW_DEBUG: Received 84 bytes flow 23/Oct/2018 10:01:31 [collect.c:854] +++ Dissecting packet 3 23/Oct/2018 10:01:31 [util.c:4802] [ZMQ] [event] { "iface": { "name": "none", "speed": 1000, "ip": "" }, "probe": { "ip": "172.21.0.200", "public_ip": "" }, "time" : 1540278091.524, "bytes": 0, "packets": 0, "avg": { "bps": 0, "pps": 0 }, "drops" : { "export_queue_too_long": 0, "too_many_flows": 0, "elk_flow_drops": 0, "sflow_pkt_sample_drops": 0 }, "timeout": { "lifetime": 120, "idle": 30 }, "zmq": { "num_flow_exports": 0, "num_exporters": 1 }, }

Just wanted to add, that now I run the nightly build repository.

simonemainardi commented 5 years ago

I've just tried to reproduce but everything seems to work as expected.

[simone@develv5 nProbe]$ sudo ./nprobe -i eno1 -n none --mysql="127.0.0.1:nprobe:nf:root:"

Then data is immediately populated:

MariaDB [nprobe]> drop table nfflow;
ERROR 1051 (42S02): Unknown table 'nfflow'
MariaDB [nprobe]> drop table nfflows;
Query OK, 0 rows affected (0.00 sec)

MariaDB [nprobe]> show tables
    -> ;
+------------------+
| Tables_in_nprobe |
+------------------+
| nfflows          |
+------------------+
1 row in set (0.00 sec)

MariaDB [nprobe]> select * from nfflows limit 1;
+-----+----------+---------+----------+---------+-----------+-------------+---------------+------------+-------------+---------------+-------------+--------+--------+---------------+----------------+---------------+---------------+
| idx | IN_BYTES | IN_PKTS | PROTOCOL | SRC_TOS | TCP_FLAGS | L4_SRC_PORT | IPV4_SRC_ADDR | INPUT_SNMP | L4_DST_PORT | IPV4_DST_ADDR | OUTPUT_SNMP | SRC_AS | DST_AS | LAST_SWITCHED | FIRST_SWITCHED | IPV6_SRC_ADDR | IPV6_DST_ADDR |
+-----+----------+---------+----------+---------+-----------+-------------+---------------+------------+-------------+---------------+-------------+--------+--------+---------------+----------------+---------------+---------------+
|   1 |      752 |       5 |        6 |      16 |        24 |          22 |    3232236257 |          0 |       52295 |    3232236162 |           0 |      0 |      0 |    1540456528 |     1540456512 |               |               |
+-----+----------+---------+----------+---------+-----------+-------------+---------------+------------+-------------+---------------+-------------+--------+--------+---------------+----------------+---------------+---------------+
1 row in set (0.00 sec)

Is seems your command line has issues. Fields are not escaped, and some others are missing the double dash. The correct line is

nprobe --zmq "tcp://127.0.0.1:5556" --zmq-probe-mode --interface ens18 --flow-templ "@NTOPNG@ %EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS" --flow-version 9 --mysql "127.0.0.1:ntop:nf:roman:PASSWORD"

Using the debug flag, you will see nProbe INSERT INTO queries at runtime as follows:

25/Oct/2018 10:36:38 [database.c:37] INSERT INTO `nfflows` (IN_BYTES, IN_PKTS, PROTOCOL, SRC_TOS, TCP_FLAGS, L4_SRC_PORT, INPUT_SNMP, L4_DST_PORT, OUTPUT_SNMP, SRC_AS, DST_AS, LAST_SWITCHED, FIRST_SWITCHED, IPV6_SRC_ADDR, IPV6_DST_ADDR) VALUES ('56', '1', '58', '0', '0', '0', '0', '0', '0', '0', '0', '1540456596', '1540456596', 'fe80::20d:b9ff:fe37:ebf8', 'ff02::2')
romeor commented 5 years ago

Hello,

You've tested exactly what is working the right way and I mentioned this one above. Try to put it into the collector mode! Okay, I've modified the line, but still no use of it: still no new database created upon starting the nprobe:

nprobe --zmq "tcp://127.0.0.1:5556" --zmq-probe-mode --interface none --collector-port 2055 --collector none --flow-templ "@NTOPNG@%EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS" --flow-version 9 --mysql "127.0.0.1:ntop:nf:roman:PASSWORD"

MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | ens | | information_schema | | mysql | | performance_schema | +--------------------+ 4 rows in set (0.00 sec)

romeor commented 5 years ago

Any updates on this issue?

romeor commented 5 years ago

installed new vm with new nprobe and ntopng installation and still no db crated at the launch.

romeor commented 5 years ago

ok more debug info here: as soon as I add the --collector-port 2055 line, nprobe won't even try to connect to DB. If I remove this line leaving --interface none nprobe connects to DB, creates it, bot does not receive any flows while still listening to 2055 and my router keeps sending them. I guess there is a bug.

romeor commented 5 years ago

Ok I went another way. I start nprobe in collector mode, but do not use the --mysql switch, otherwise, I use -F mysql in the ntopng configuration file.

Please fix nprobe. Reach me if you need more information.

simonemainardi commented 5 years ago

Thank you for reporting. I have verified that there was an issue with MySQL export when nProbe was working in collector mode. The issue is now fixed. A new 8.7 build will be available in one hour.

romeor commented 5 years ago

confirmed: working.