No Data Available -- Windows -- Netflow

jmunty commented 8 years ago

Hi I've had a really good go at this, but can't seem to make it work.

Windows 2012 R2 nProbeWin-x64-7.3.160319.zip ntopng-2.3.160319-x64.zip

I've read both manuals over a couple of times

The nprobe manual says that for using ntopng with nprobe you should use the following config: ntop Configuration ntopng -i tcp://127.0.0.1:5556 nProbe Configuration

nprobe --zmq "tcp://*:5556" -i eth1 -n none (probe mode)
nprobe --zmq "tcp://*:5556" -i none -n none --collector-port 2055 (sFlow/NetFlow collector mode)

I then did the following

ran the installation files contained in the zip.
confirmed that netflow traffic was indeed being received on port 2055
checked that no other services were listening on port 2055 or 5556

then i ran the following commands (as administrator)

nprobe /i  nprobe --zmq "tcp://*:5556" -i none -n none --collector-port 2055
ntopng /i ntopng -i tcp://127.0.0.1:5556

then i

started the redis service
started ntopng and nprobe services

on logging into the ntopng GUI i got the "no data is available"

Here is a list of the other things i have tried:

Remove the ntopng/nprobe services and re-add them using the /r switch
Tried npobe in debug mode (using --debug switch) and i can see that nprobe sees the netflow data, but when i ctrl-c to end it, it tells me that although packets were collected, none were processed..... Also no data appears in the web GUI

nprobe /c nprobe -i none -n none --collector-port 2055 --zmq "tcp://*:5556" --debug

Tried nprobe in the mode to spit the data out to file. This works fine, the files are populated. Although still no data appears in the web GUI

nprobe /c nprobe -P D:\temp\nprobe -D t -i none -n none --collector-port 2055 --zmq "tcp://*:5556" --debug

Tried changing the order of the --zmq switch to the start or to the end of the nprobe command
Tried running both nprobe and ntopng from the console... still no data.

ntopng /c ntopng -i tcp://127.0.0.1:5556
nprobe /c nprobe -i none -n none --collector-port 2055 --zmq "tcp://*:5556" --debug

Tried running ntopng in verbose mode... doesn't give any new info
Tried changing around the order of starting ntopng before nprobe and vice versa
Tried changing the zmq port to 5557 on both nprobe and ntopng
Restarted server
Tried running ntop in service mode while nprobe in command line mode & vice versa
Tried deleting the ntopng files in c:\users\username\documents each time i restarted the ntopng service or reran it from the command line with /c
Tried running nprobe with different interfaces in the -i switch. using "1" in debug mode provides the same data as "none" (i.e. it picks up the netflow traffic). Using "2" provides no netflow data in debug mode

nprobe /c nprobe -P D:\temp\nprobe -D t -i 1 -n none --collector-port 2055 --zmq "tcp://*:5556" --debug

Tried running the /c command without the second specification of nprobe in the argument:

nprobe /c -i none -n none --collector-port 2055 --zmq "tcp://*:5556" --debug

Waited up to 10mins between each try to see if data fetching was delayed
Ran sysinternals procmon. For nprobe process i see the receipt of the netflow packets, but no activity on port 5556. For ntopng process i see TCP Reconnect events on port 5556 going to high order ports like 61576... but no constant traffic on port 5556...

things i haven't tried...

both the ntopng and nprobe service are registered to run as Local System. is this ok?
previous versions... should i try a previous version?
in the verbose mode of ntopng should there be info about whether its picking up data from zmq?

Questions: Is there a way to see if zmq is actually working on Windows?? As i cant verify the this part of the link......

jmunty commented 8 years ago

oh yeah i also tried deleting the nprobe temp files in C:\Windows\Temp\0 and restarting ntop but that didnt work.

Also the NTOP GUI did work once for a couple of minutes. I stopped it thinking that i was nearly done with the configuration. but i've never been able to get it working this way since... The combination i used was as below:

nprobe /i  nprobe -P D:\temp\nprobe -D t  -i 1 -n none --collector-port 2055 --zmq "tcp://*:5556"
ntopng /i ntopng -i tcp://127.0.0.1:5556

jmunty commented 8 years ago

btw i am using a trial licence for both ntopng and nprobe. the licence for nprobe is in c:\program files\nprobe and is recognized when nprobe starts up

the licence for ntopng i inserted in the GUI and it seems to stay there even when i create/delete the service or run it from command line

I also tried typing in all commands by hand rather than copy/paste

This is what ntopng looks like when started from command line

and what it looks like when shutting down

and here is selected parts of the log:

Starting ntopg
Running ntopng.
29/Apr/2016 22:10:44 [Prefs.cpp:831] Logging into ----removed-----
29/Apr/2016 22:10:44 [Ntop.cpp:980] Setting local networks to 127.0.0.0/8
29/Apr/2016 22:10:44 [AddressTree.cpp:139] Rule 127.0.0.0/8
29/Apr/2016 22:10:44 [Redis.cpp:106] Successfully connected to redis 127.0.0.1:6379@0
29/Apr/2016 22:10:44 [NtopPro.cpp:116] [LICENSE] Read license from Redis [----removed-----]
29/Apr/2016 22:10:44 [NtopPro.cpp:153] WARNING: [LICENSE] Your license will expire on Thu May 26 05:53:27 2016
29/Apr/2016 22:10:44 [L7Policer.cpp:80] Created policer 0 with no shaping max rate
29/Apr/2016 22:10:44 [L7Policer.cpp:80] Created policer 1 with no shaping max rate
29/Apr/2016 22:10:44 [L7Policer.cpp:80] Created policer 2 with no shaping max rate
29/Apr/2016 22:10:44 [L7Policer.cpp:80] Created policer 3 with no shaping max rate
29/Apr/2016 22:10:44 [L7Policer.cpp:80] Created policer 4 with no shaping max rate
29/Apr/2016 22:10:44 [L7Policer.cpp:80] Created policer 5 with no shaping max rate
29/Apr/2016 22:10:44 [L7Policer.cpp:80] Created policer 6 with no shaping max rate
29/Apr/2016 22:10:44 [L7Policer.cpp:80] Created policer 7 with no shaping max rate
29/Apr/2016 22:10:44 [L7Policer.cpp:80] Created policer 8 with no shaping max rate
29/Apr/2016 22:10:44 [L7Policer.cpp:80] Created policer 9 with no shaping max rate
29/Apr/2016 22:10:44 [Ntop.cpp:1199] Registered interface tcp://127.0.0.1:5556 [id: 0]
29/Apr/2016 22:10:44 [Ntop.cpp:1212] Registered interface view tcp://127.0.0.1:5556 [id: 0]
29/Apr/2016 22:10:44 [HTTPserver.cpp:464] HTTPS Disabled: missing SSL certificate c:\Program Files\ntopng\httpdocs/ssl/ntopng-cert.pem
29/Apr/2016 22:10:44 [HTTPserver.cpp:466] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL.
29/Apr/2016 22:10:44 [HTTPserver.cpp:509] Web server dirs [c:\Program Files\ntopng\httpdocs][c:\Program Files\ntopng\scripts]
29/Apr/2016 22:10:44 [HTTPserver.cpp:512] HTTP server listening on port 3000
29/Apr/2016 22:10:44 [main.cpp:295] Working directory: ----removed-----
29/Apr/2016 22:10:44 [main.cpp:297] Scripts/HTML pages directory: c:\Program Files\ntopng
29/Apr/2016 22:10:44 [Ntop.cpp:258] Welcome to ntopng x64 v.2.3.160306 - (C) 1998-16 ntop.org
29/Apr/2016 22:10:44 [Ntop.cpp:263] Built on Windows
29/Apr/2016 22:10:44 [PeriodicActivities.cpp:53] Started periodic activities loop...
29/Apr/2016 22:10:44 [RuntimePrefs.cpp:34] Dumping alerts into syslog
29/Apr/2016 22:10:44 [PeriodicActivities.cpp:91] Starting script c:\Program Files\ntopng\scripts\callbacks/second.lua
29/Apr/2016 22:10:44 [NtopPro.cpp:233] [LICENSE] ntopng systemId: ----removed-----
29/Apr/2016 22:10:44 [NtopPro.cpp:238] [LICENSE] ntopng is starting in demo mode
29/Apr/2016 22:10:44 [Lua.cpp:2654] ntop_get_dirs() called
29/Apr/2016 22:10:44 [Lua.cpp:2576] ntop_is_pro() called
29/Apr/2016 22:10:44 [Lua.cpp:2654] ntop_get_dirs() called
29/Apr/2016 22:10:44 [NetworkInterface.cpp:1435] Started packet polling on interface tcp://127.0.0.1:5556 [id: 0]...
29/Apr/2016 22:10:44 [CollectorInterface.cpp:104] Collecting flows on tcp://127.0.0.1:5556 [ntopng->nprobe]
29/Apr/2016 22:10:44 [Lua.cpp:2278] ntop_http_get_prefix() called
29/Apr/2016 22:10:44 [Lua.cpp:2278] ntop_http_get_prefix() called
29/Apr/2016 22:10:44 [Lua.cpp:2278] ntop_http_get_prefix() called
29/Apr/2016 22:10:44 [Lua.cpp:2278] ntop_http_get_prefix() called
29/Apr/2016 22:10:44 [Lua.cpp:2576] ntop_is_pro() called
29/Apr/2016 22:10:44 [Lua.cpp:205] ntop_get_interface_names() called
29/Apr/2016 22:10:44 [Lua.cpp:255] ntop_select_interface() called

lucaderi commented 8 years ago

@jmunty It looks like the nProbe package was not properly built. I am uploading a new nprobe package in http://packages.ntop.org/Windows/ that should fix the issue.

jmunty commented 8 years ago

Hi Luca Many thanks its all working now! I can see in the output in debug mode that nprobe is bringing up the zmq service now. I only just noticed that the windows installer creates the services for you. Might be worth changing that section in the nprobe manual as people who read it will probably think they still have to create the service themselves

jmunty commented 8 years ago

Hi Luca,

TLDR - is there something missing from the ntopng windows binary? Maybe something to do with the zmq listener service? I can't see stuff being received into ntopng.

I thought it was solved but its not yet.

What i did:

uninstalled redis
removed old ntopng and nprobe services with /r switch
deleted ntopng and nprobe program folders
deleted all webdirs in C:\Windows\Temp\0 and c:\users\username\Documents\0
restarted machine
installed ntopng-2.3.160319-x64
installed nProbeWin-x64-7.3.160319-2
started redis
started ntopng and nprobe services that were automatically created by installer

This seemed to work but the ntopng service is not pulling netflow data from nprobe over zmq. It is just pulling in whatever was on my main NIC - straight packet caputure.

I confirmed this by:

seeing that only traffic to/from the nprobe/ntopng server was included in the list of flows
turning off the nprobe service, ntopng is still receiving traffic

Now on the plus side, it looks like the new nprobe binary you uploaded is working fine,.. i now see reports in the debug log that the zmq service has been started which i didnt see before:

particularly

Initializing ZMQ as server

Running nProbe for Windows.
I
04/May/2016 01:25:57 [nprobe.c:3265] ERROR: Invalid nProbe license (nprobe.license) [Missing license file]
04/May/2016 01:25:57 [nprobe.c:3272] ERROR: *****************************************************
04/May/2016 01:25:57 [nprobe.c:3273] ERROR: **                                                 **
04/May/2016 01:25:57 [nprobe.c:3274] ERROR: **  Switching to DEMO MODE (missing valid license) **
04/May/2016 01:25:57 [nprobe.c:3275] ERROR: **                                                 **
04/May/2016 01:25:57 [nprobe.c:3276] ERROR: **  Purchase your nProbe license at                **
04/May/2016 01:25:57 [nprobe.c:3277] ERROR: **       https://shop.ntop.org/                    **
04/May/2016 01:25:57 [nprobe.c:3278] ERROR: **                                                 **
04/May/2016 01:25:57 [nprobe.c:3279] ERROR: *****************************************************
04/May/2016 01:25:57 [nprobe.c:6981] ERROR: ***************************************************************
04/May/2016 01:25:57 [nprobe.c:6982] ERROR: * NOTE: This is a DEMO version limited to 25000 flows export.  *
04/May/2016 01:25:57 [nprobe.c:6983] ERROR: ***************************************************************
04/May/2016 01:25:57 [nprobe.c:4687] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
04/May/2016 01:25:57 [nprobe.c:4690] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
04/May/2016 01:25:57 [nprobe.c:4784] Welcome to nProbe Pro v.7.3.160315 ($Revision: 4384 $) for Windows 
04/May/2016 01:25:57 [nprobe.c:4794] Running on Windows
04/May/2016 01:25:57 [nprobe.c:4805] [LICENSE] nProbe SystemId: ----removed-----
04/May/2016 01:25:57 [nprobe.c:6999] Welcome to nProbe v.7.3.160315 for Windows
04/May/2016 01:25:57 [cache.c:1239] init_lru_cache(max_size=16384)
04/May/2016 01:25:57 [cache.c:1239] init_lru_cache(max_size=16384)
04/May/2016 01:25:57 [plugin.c:1022] 0 plugin(s) enabled
04/May/2016 01:25:57 [nprobe.c:6583] Non IPv4/v6 traffic is discarded according to the template
04/May/2016 01:25:57 [nprobe.c:5356] Using packet capture length 128
04/May/2016 01:25:57 [nprobe.c:7172] IPv6 traffic will NOT be exported/accounted by this probe
04/May/2016 01:25:57 [nprobe.c:7173] due to configuration options (e.g. use NetFlow v9)
04/May/2016 01:25:57 [nprobe.c:7218] Flows ASs will not be computed (missing GeoIP support)
04/May/2016 01:25:57 [nprobe.c:7301] Not capturing packet from interface (collector mode)
04/May/2016 01:25:57 [util.c:4030] Initializing ZMQ as server
04/May/2016 01:25:57 [util.c:4073] Succesfully created ZMQ endpoint tcp://*:5556
04/May/2016 01:25:57 [collect.c:145] Flow collector listening on port 2055 (IPv4/v6)
04/May/2016 01:25:57 [nprobe.c:7402] WARNING: *****************************************
04/May/2016 01:25:57 [nprobe.c:7403] WARNING: ** You're running nprobe in DEBUG mode **
04/May/2016 01:25:57 [nprobe.c:7404] WARNING: *****************************************
04/May/2016 01:25:57 [nprobe.c:7514] nProbe started successfully
04/May/2016 01:25:57 [collect.c:1764] NETFLOW_DEBUG: Received 1448 bytes flow
04/May/2016 01:25:58 [collect.c:1764] NETFLOW_DEBUG: Received 1428 bytes flow
04/May/2016 01:25:58 [collect.c:1764] NETFLOW_DEBUG: Received 1472 bytes flow
04/May/2016 01:25:58 [collect.c:1764] NETFLOW_DEBUG: Received 1448 bytes flow
04/May/2016 01:25:58 [collect.c:1764] NETFLOW_DEBUG: Received 1452 bytes flow
04/May/2016 01:25:58 [collect.c:1764] NETFLOW_DEBUG: Received 1452 bytes flow
04/May/2016 01:25:59 [collect.c:1764] NETFLOW_DEBUG: Received 1424 bytes flow
.
.
.
.
04/May/2016 01:26:18 [collect.c:1764] NETFLOW_DEBUG: Received 1472 bytes flow
04/May/2016 01:26:18 [collect.c:1764] NETFLOW_DEBUG: Received 1452 bytes flow
04/May/2016 01:26:20 [collect.c:1764] NETFLOW_DEBUG: Received 1464 bytes flow
04/May/2016 01:26:21 [collect.c:1764] NETFLOW_DEBUG: Received 1420 bytes flow
04/May/2016 01:26:22 [collect.c:1764] NETFLOW_DEBUG: Received 1472 bytes flow
04/May/2016 01:26:22 [cache.c:1224] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
04/May/2016 01:26:22 [nprobe.c:429] Received shutdown request... [signal: 2]
04/May/2016 01:26:25 [cache.c:1224] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
04/May/2016 01:26:25 [cache.c:1284] free_lru_cache()
04/May/2016 01:26:25 [cache.c:1284] free_lru_cache()
04/May/2016 01:26:25 [nprobe.c:2595] Processed packets: 0 (max bucket search: 0)
04/May/2016 01:26:25 [nprobe.c:2578] Fragment queue length: 0
04/May/2016 01:26:25 [nprobe.c:2604] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
04/May/2016 01:26:25 [nprobe.c:2611] Flow collection: [collected pkts: 46][processed flows: 0]
04/May/2016 01:26:25 [nprobe.c:2614] Flow drop stats:   [0 bytes/0 pkts][0 flows]
04/May/2016 01:26:25 [nprobe.c:2619] Total flow stats:  [0 bytes/0 pkts][0 flows/0 pkts sent]

So now for ntopng...

Starting ntopg
Running ntopng.
04/May/2016 01:10:34 [Prefs.cpp:831] Logging into C:\Users\username\Documents\ntopng.log
04/May/2016 01:10:34 [Ntop.cpp:980] Setting local networks to 127.0.0.0/8
04/May/2016 01:10:34 [AddressTree.cpp:139] Rule 127.0.0.0/8
04/May/2016 01:10:34 [Redis.cpp:106] Successfully connected to redis 127.0.0.1:6379@0
04/May/2016 01:10:34 [NtopPro.cpp:116] [LICENSE] Read license from Redis [--removed--]
04/May/2016 01:10:34 [NtopPro.cpp:153] WARNING: [LICENSE] Your license will expire on Thu May 26 05:53:27 2016
04/May/2016 01:10:34 [L7Policer.cpp:80] Created policer 0 with no shaping max rate
04/May/2016 01:10:34 [L7Policer.cpp:80] Created policer 1 with no shaping max rate
04/May/2016 01:10:34 [L7Policer.cpp:80] Created policer 2 with no shaping max rate
04/May/2016 01:10:34 [L7Policer.cpp:80] Created policer 3 with no shaping max rate
04/May/2016 01:10:34 [L7Policer.cpp:80] Created policer 4 with no shaping max rate
04/May/2016 01:10:34 [L7Policer.cpp:80] Created policer 5 with no shaping max rate
04/May/2016 01:10:34 [L7Policer.cpp:80] Created policer 6 with no shaping max rate
04/May/2016 01:10:34 [L7Policer.cpp:80] Created policer 7 with no shaping max rate
04/May/2016 01:10:34 [L7Policer.cpp:80] Created policer 8 with no shaping max rate
04/May/2016 01:10:34 [L7Policer.cpp:80] Created policer 9 with no shaping max rate
04/May/2016 01:10:34 [Ntop.cpp:1199] Registered interface tcp://127.0.0.1:5556 [id: 0]
04/May/2016 01:10:34 [Ntop.cpp:1212] Registered interface view tcp://127.0.0.1:5556 [id: 0]
04/May/2016 01:10:34 [HTTPserver.cpp:464] HTTPS Disabled: missing SSL certificate c:\Program Files\ntopng\httpdocs/ssl/ntopng-cert.pem
04/May/2016 01:10:34 [HTTPserver.cpp:466] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL.
04/May/2016 01:10:34 [HTTPserver.cpp:509] Web server dirs [c:\Program Files\ntopng\httpdocs][c:\Program Files\ntopng\scripts]
04/May/2016 01:10:34 [HTTPserver.cpp:512] HTTP server listening on port 3000
04/May/2016 01:10:34 [main.cpp:295] Working directory: C:\Users\username\Documents
04/May/2016 01:10:34 [main.cpp:297] Scripts/HTML pages directory: c:\Program Files\ntopng
04/May/2016 01:10:34 [Ntop.cpp:258] Welcome to ntopng x64 v.2.3.160306 - (C) 1998-16 ntop.org
04/May/2016 01:10:34 [Ntop.cpp:263] Built on Windows
04/May/2016 01:10:34 [PeriodicActivities.cpp:53] Started periodic activities loop...
04/May/2016 01:10:34 [RuntimePrefs.cpp:34] Dumping alerts into syslog
04/May/2016 01:10:34 [PeriodicActivities.cpp:91] Starting script c:\Program Files\ntopng\scripts\callbacks/second.lua
04/May/2016 01:10:34 [NtopPro.cpp:233] [LICENSE] ntopng systemId: 2422883391-76066acf
04/May/2016 01:10:34 [NtopPro.cpp:238] [LICENSE] ntopng is starting in demo mode
04/May/2016 01:10:34 [PeriodicActivities.cpp:91] Starting script c:\Program Files\ntopng\scripts\callbacks/daily.lua
04/May/2016 01:10:34 [Lua.cpp:2654] ntop_get_dirs() called
04/May/2016 01:10:34 [Lua.cpp:2654] ntop_get_dirs() called
04/May/2016 01:10:34 [Lua.cpp:2576] ntop_is_pro() called
04/May/2016 01:10:34 [NetworkInterface.cpp:1435] Started packet polling on interface tcp://127.0.0.1:5556 [id: 0]...
04/May/2016 01:10:34 [Lua.cpp:2576] ntop_is_pro() called
04/May/2016 01:10:34 [Lua.cpp:2654] ntop_get_dirs() called
04/May/2016 01:10:34 [Lua.cpp:2654] ntop_get_dirs() called
04/May/2016 01:10:34 [Lua.cpp:2654] ntop_get_dirs() called
.
.
.
04/May/2016 01:10:34 [Lua.cpp:3220] ntop_stats_delete_hour_older_than() called
04/May/2016 01:10:34 [Lua.cpp:3258] ntop_stats_delete_day_older_than() called
04/May/2016 01:10:34 [Lua.cpp:3398] ntop_stats_get_samplings_of_hours_from_epoch() called
04/May/2016 01:10:34 [Lua.cpp:2015] ntop_rrd_update(C:\Users\username\Documents\0\rrd\bytes.rrd) N:0
04/May/2016 01:10:34 [CollectorInterface.cpp:104] Collecting flows on tcp://127.0.0.1:5556 [ntopng->nprobe]
04/May/2016 01:10:34 [Lua.cpp:607] ntop_is_windows() called
04/May/2016 01:10:34 [Lua.cpp:670] ntop_list_dir_files() called
04/May/2016 01:10:34 [Lua.cpp:2654] ntop_get_dirs() called
04/May/2016 01:10:34 [Lua.cpp:2654] ntop_get_dirs() called
.
.
.
04/May/2016 01:12:03 [Lua.cpp:3945] ntop_lua_http_print() called
04/May/2016 01:12:03 [Lua.cpp:3945] ntop_lua_http_print() called
04/May/2016 01:12:03 [main.cpp:37] Shutting down...
04/May/2016 01:12:04 [CollectorInterface.cpp:183] [159] { "if.name": "none", "if.speed": 1000, "if.ip": "", "probe.ip": "127.0.0.1", "probe.public_ip": "--removed--", "time" : 1462288324, "bytes": 0, "packets": 0 }
04/May/2016 01:12:05 [CollectorInterface.cpp:183] [159] { "if.name": "none", "if.speed": 1000, "if.ip": "", "probe.ip": "127.0.0.1", "probe.public_ip": "--removed--", "time" : 1462288325, "bytes": 0, "packets": 0 }
04/May/2016 01:12:05 [ProtoStats.cpp:35] [IPv4]  0 B/0.00 Packets
04/May/2016 01:12:05 [ProtoStats.cpp:35] [IPv6]  0 B/0.00 Packets
04/May/2016 01:12:05 [ProtoStats.cpp:35] [ARP]   0 B/0.00 Packets
04/May/2016 01:12:05 [ProtoStats.cpp:35] [MPLS]  0 B/0.00 Packets
04/May/2016 01:12:05 [ProtoStats.cpp:35] [Other] 0 B/0.00 Packets
04/May/2016 01:12:06 [Ntop.cpp:1238] Interface tcp://127.0.0.1:5556 [running: 0]
04/May/2016 01:12:06 [Profiles.cpp:36] Destroying Profiles
04/May/2016 01:12:07 [HTTPserver.cpp:524] HTTP server terminated
04/May/2016 01:12:07 [AddressResolution.cpp:54] Address resolution stats [0 resolved][0 failures]

So basically it looks like it's running but not picking up traffic properly from zmq??

lucaderi commented 8 years ago

@jmunty How did you start ntopng exactly? I see you have a view interface. Please on the ntopng web gui set the interface to the collector interface.

simonemainardi commented 8 years ago

@jmunty any news?

jmunty commented 8 years ago

Guys, sorry for the late reply .- been off a few days. The view interface is set to the collector interface. It doesn't work.

What tools are there to see:

what zmq traffic is being pushed by nprobe
what zmq traffic is being consumed by ntopng?

Also,

what should i see in the ntopng log when netflow data is being successfully received/consumed?
when i quit nprobe, why does it say [collected pkts: 71][processed flows: 0] -- what does this mean exactly? does it mean that nprobe couldnt connect to ntopng? and hence nprobe couldn't process packets?

here is the screenshot showing the GUI during collection -- no data...

finally, i see this all the time in the ntopng log.... i assume it means that there are N:0 updates to the RRD file?

11/May/2016 22:58:52 [Lua.cpp:2015] ntop_rrd_update(C:\Users\username\Documents\0\rrd\bytes.rrd) N:0
11/May/2016 22:58:52 [Lua.cpp:607] ntop_is_windows() called
11/May/2016 22:58:52 [Lua.cpp:523] ntop_get_file_dir_exists() called
11/May/2016 22:58:52 [Lua.cpp:2015] ntop_rrd_update(C:\Users\username\Documents\0\rrd\packets.rrd) N:0
11/May/2016 22:58:52 [Lua.cpp:607] ntop_is_windows() called
11/May/2016 22:58:52 [Lua.cpp:523] ntop_get_file_dir_exists() called
11/May/2016 22:58:52 [Lua.cpp:2015] ntop_rrd_update(C:\Users\username\Documents\0\rrd\drops.rrd) N:0
11/May/2016 22:58:52 [Lua.cpp:607] ntop_is_windows() called
11/May/2016 22:58:52 [Lua.cpp:523] ntop_get_file_dir_exists() called
11/May/2016 22:58:52 [Lua.cpp:2015] ntop_rrd_update(C:\Users\username\Documents\0\rrd\num_hosts.rrd) N:0
11/May/2016 22:58:52 [Lua.cpp:607] ntop_is_windows() called
11/May/2016 22:58:52 [Lua.cpp:523] ntop_get_file_dir_exists() called
11/May/2016 22:58:52 [Lua.cpp:2015] ntop_rrd_update(C:\Users\username\Documents\0\rrd\num_flows.rrd) N:0
11/May/2016 22:58:52 [Lua.cpp:607] ntop_is_windows() called
11/May/2016 22:58:52 [Lua.cpp:523] ntop_get_file_dir_exists() called

i also see this regularly in the record as well...

11/May/2016 22:58:52 [Lua.cpp:607] ntop_is_windows() called
11/May/2016 22:58:52 [Lua.cpp:523] ntop_get_file_dir_exists() called
11/May/2016 22:58:52 [Lua.cpp:2015] ntop_rrd_update(C:\Users\username\Documents\0\rrd\num_http_hosts.rrd) N:0
11/May/2016 22:58:52 [CollectorInterface.cpp:183] [159] { "if.name": "none", "if.speed": 1000, "if.ip": "", "probe.ip": "127.0.0.1", "probe.public_ip": "some.ip.address", "time" : 1462971532, "bytes": 0, "packets": 0 }
11/May/2016 22:58:52 [HTTPserver.cpp:223] [HTTP] Session f5b2b8ea5cc9066cb5c04b4eb9dd0d3a is OK: extended for 43200 sec
11/May/2016 22:58:52 [HTTPserver.cpp:353] [HTTP] /lua/network_load.lua

jmunty commented 8 years ago

also... when i run a windows process monitor (procmon) i see very little activity from nprobe/ntopng on port tcp 5556. Most of the activity seems to run on: nprobe: localhost:5905 <=> localhost:58319-58322 ntopng: localhost:6379 <=> localhost:58488

not sure if that helps at all...

jmunty commented 8 years ago

Hi Guys do you have any news on this issue at all? I posted on github some days ago

jmunty commented 8 years ago

Hi

a little more info i saw today. ....

When i run Wireshark on the data i can see a few things. Firstly i can see the template packets coming through... and the templates themselves look like this:

When i drill down into wireshark to see the flows that are encapsulated in a single packet (for template 263) i can see that the data appears correctly. I also see that there are source/destination IPs in the data that are NOT the netflow device, nor the PC i'm using to collect the netflow data.

So i guess this proves that netflow is configured correctly on the netflow device and is being captured correctly on the PC.

Now when i do a file dump command like this: nprobe /c nprobe -P D:\temp\nprobe -D t -i none -n none --collector-port 2055 --zmq "tcp://*:5556"

The dump file output shows ONLY traffic going to/from my PC.... strange...?

Should the dump file actually dump the full netflow traffic or should it ONLY show the traffic that has my PC as source/destination?

Finally, again

... is there any way to verify that what is coming out of the nprobe on the zmq protocol is the correct netflow info?
Is there any way to verify that ntopng is ingesting the netflow data correctly

m00tpoint commented 8 years ago

I am seeing this behavior on a new install on CentOS as well, version v.7.2.160512 for both NProbe and NTop.

nprobe -n none -i none -3 6343 --zmq "tcp://*:5555" ntopng -i tcp://127.0.0.1:5555

nprobe verbose 2 and debug clearly show numerous inbound sflow packets arriving and being decoded, but 0 packets are sent to NTOP over tcp 5555. If I dump to disk with no nested directories, I get a 204-byte file with column headers and no data. When I terminate nprobe, I get:

19/May/2016 14:58:17 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec] 19/May/2016 14:58:17 [nprobe.c:386] Received shutdown request... [signal: 15] 19/May/2016 14:58:18 [nprobe.c:4716] nProbe is shutting down... 19/May/2016 14:58:18 [nprobe.c:4752] Exporting pending buckets... 19/May/2016 14:58:18 [engine.c:2673] About to flush hash (threadId 0) 19/May/2016 14:58:18 [engine.c:2675] Completed hash walk (thread 0) 19/May/2016 14:58:18 [nprobe.c:4758] Waiting to export queued buckets... [queue len=42] 19/May/2016 14:58:19 [nprobe.c:4773] Pending buckets have been exported... 19/May/2016 14:58:21 [engine.c:3293] Export thread terminated [exportQueue=0] 19/May/2016 14:58:21 [nprobe.c:4839] Flushing queued flows... 19/May/2016 14:58:21 [nprobe.c:4842] Freeing memory... 19/May/2016 14:58:21 [plugin.c:277] Terminating plugins. 19/May/2016 14:58:21 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec] 19/May/2016 14:58:21 [nprobe.c:4934] Still allocated 0 hash buckets ****_19/May/2016 14:58:21 [nprobe.c:2457] Processed packets: 601 (max bucket search: 1) 19/May/2016 14:58:21 [nprobe.c:2440] Fragment queue length: 0 19/May/2016 14:58:21 [nprobe.c:2466] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent] 19/May/2016 14:58:21 [nprobe.c:2473] Flow collection: [collected pkts: 4387][processed flows: 0] 19/May/2016 14:58:21 [nprobe.c:2476] Flow drop stats: [0 bytes/0 pkts][0 flows] 19/May/2016 14:58:21 [nprobe.c:2481] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent] _***19/May/2016 14:58:21 [nprobe.c:4947] Cleaning globals 19/May/2016 14:58:21 [nprobe.c:4967] nProbe terminated.

lucaderi commented 8 years ago

@m00tpoint Can you please send me via email a pcap containing flows + templates so we can check what we need to change in nProbe in order to support your device?

m00tpoint commented 8 years ago

Sent, and thanks!

simonemainardi commented 8 years ago

@jmunty please try and install the latest windows build and report back. Thank you

jmunty commented 8 years ago

Hi Simone Great thanks for that – will try it out today!

jmunty commented 8 years ago

Hi Simone

Had a quick shot at it yesterday but couldn’t get it to work.

Pretty tied up today and tomorrow so give me till around Tuesday and I should have some results

jmunty commented 8 years ago

HI Simone Thanks for your work on this. I still can't get it to work. I’m going to try the linux version shortly instead. Thanks again!

jmunty commented 8 years ago

Hi Simone I’m still in the process of getting a test system setup so I can send you the traffic. Would it be enough though if I send you just the template files from packet captures on my production system? Would that help with the troubleshooting?

lucaderi commented 8 years ago

@jmunty Sorry for the delay but we have been busy with the release of nprobe 7.4 where we have made quite some fixes. Did you try that perhaps? What version of ntopng are you using? The problem is a bit odd and IMHO it should work.

jmunty commented 8 years ago

Hi many thanks! Will check this out :)

Calvin Chiang | SaaS Operations Engineer P: +47 9540 5384 M: +47 2150 2562

From: Luca Deri [mailto:notifications@github.com] Sent: 24. juni 2016 21:34 To: ntop/ntopng ntopng@noreply.github.com Cc: Calvin Chiang Calvin.Chiang@confirmit.com; Mention mention@noreply.github.com Subject: Re: [ntop/ntopng] No Data Available -- Windows -- Netflow (#542)

@jmunty https://github.com/jmunty Sorry for the delay but we have been busy with the release of nprobe 7.4 where we have made quite some fixes. Did you try that perhaps? What version of ntopng are you using? The problem is a bit odd and IMHO it should work.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ntop/ntopng/issues/542#issuecomment-228440871 , or mute the thread https://github.com/notifications/unsubscribe/AR3nYSlGq4r1M-y-kd6ALulF1cEP1R--ks5qPDE1gaJpZM4ISwK4 . https://github.com/notifications/beacon/AR3nYVa11fV_R2JNonreMRFV0aPUJ00cks5qPDE1gaJpZM4ISwK4.gif

surajit26 commented 8 years ago

Hi,

I have configured ntopng 2.4.160627 - Community Edition on windows 2008 server (refer attached snapshot). I have enabled 6 Cisco routers interfaces as follows to view netflow for local as well as for other remote 5 locations: R1#configure terminal R1(config)#interface f0/1 R1(config-if)#ip route-cache flow R1(config-if)#ip flow ingress R1(config-if)#ip flow egress R1(config-if)#exit R1(config)#ip flow-export source f0/1 R1(config)#ip flow-cache timeout active 60 R1(config)#ip flow-cache timeout inactive 120 R1(config)#ip flow-export version 9 R1(config)#ip flow-export destination 192.168.22.10 2055

After installation the extracted zip file, I had started redis service and ntopng services. Opened URL with http://192.168.22.10:3000/ however, I am getting following error. Subsequently, I am not getting any information for routers from host list, any information about layer 7 protocols. I have installed cacti for network bandwidth monitoring and further deep analyzing of bandwidth consumption I was trying with ntopng but failed. Please suggest.

24/Jul/2016 08:35:01 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\10\ICMP.rrd: illegal attempt to update using time 1469374501 when last update time is 1469374501 (minimum one second step)] 24/Jul/2016 08:30:01 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\10\ICMP.rrd: illegal attempt to update using time 1469374201 when last update time is 1469374201 (minimum one second step)] 24/Jul/2016 08:25:01 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\10\ICMP.rrd: illegal attempt to update using time 1469373901 when last update time is 1469373901 (minimum one second step)] 24/Jul/2016 08:20:02 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\10\ICMP.rrd: illegal attempt to update using time 1469373602 when last update time is 1469373602 (minimum one second step)] 24/Jul/2016 08:15:02 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\230\ICMP.rrd: illegal attempt to update using time 1469373302 when last update time is 1469373302 (minimum one second step)] 24/Jul/2016 08:10:02 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\230\ICMP.rrd: illegal attempt to update using time 1469373002 when last update time is 1469373002 (minimum one second step)] 24/Jul/2016 08:05:02 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\230\ICMP.rrd: illegal attempt to update using time 1469372702 when last update time is 1469372702 (minimum one second step)] 24/Jul/2016 08:00:02 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\230\ICMP.rrd: illegal attempt to update using time 1469372402 when last update time is 1469372402 (minimum one second step)] 24/Jul/2016 07:55:02 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\10\ICMP.rrd: illegal attempt to update using time 1469372102 when last update time is 1469372102 (minimum one second step)] 24/Jul/2016 07:50:02 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\230\ICMP.rrd: illegal attempt to update using time 1469371802 when last update time is 1469371802 (minimum one second step)] 24/Jul/2016 07:45:02 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\230\ICMP.rrd: illegal attempt to update using time 1469371502 when last update time is 1469371502 (minimum one second step)] 24/Jul/2016 07:40:02 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\10\ICMP.rrd: illegal attempt to update using time 1469371202 when last update time is 1469371202 (minimum one second step)] 24/Jul/2016 07:35:02 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\230\ICMP.rrd: illegal attempt to update using time 1469370902 when last update time is 1469370902 (minimum one second step)] 24/Jul/2016 07:34:47 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:34:44 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:34:44 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:33:21 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:33:21 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:32:59 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_flows_sankey.lua][C:\Program Files\ntopng\scripts\lua\iface_flows_sankey.lua:24: bad argument #1 to 'pairs' (table expected, got userdata)] 24/Jul/2016 07:30:01 [Lua.cpp:4755] WARNING: Script failure [C:\Program Files\ntopng\scripts\callbacks/minute.lua][C:\Program Files\ntopng\scripts\callbacks/minute.lua:222: C:\Windows\Temp\0\rrd\192\168\22\10\ICMP.rrd: illegal attempt to update using time 1469370601 when last update time is 1469370601 (minimum one second step)] 24/Jul/2016 07:28:50 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:28:47 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:28:47 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:26:30 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:26:27 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:26:22 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:26:19 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:26:16 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:26:13 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:26:10 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:26:07 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] 24/Jul/2016 07:26:04 [Lua.cpp:5134] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua][C:\Program Files\ntopng\scripts\lua\iface_local_stats.lua:24: attempt to index a nil value] ntopng error 1

lucaderi commented 8 years ago

@jmunty What are the command line options you have used to start ntopng?

surajit26 commented 8 years ago

Hi Lucaderi,

Any update on this please (refer above attached snapshot and host stat output log.

Thanks, Surajit

lucaderi commented 8 years ago

@surajit26 I am lost on this issue. Too many cross comments, and no clear idea of what you need and what is the problem about. @simonemainardi Can you reproduce the problem?

surajit26 commented 8 years ago

Hi Lucaderi,

Thanks for your response.

I have enabled six Cisco routers as follows from different locations to view netflow: R1#configure terminal R1(config)#interface GigabitEthernet0/1 R1(config-if)#ip route-cache flow R1(config-if)#ip flow ingress R1(config-if)#ip flow egress R1(config-if)#exit R1(config)#ip flow-export source GigabitEthernet0/1 R1(config)#ip flow-cache timeout active 60 R1(config)#ip flow-cache timeout inactive 120 R1(config)#ip flow-export version 9 R1(config)#ip flow-export destination 192.168.22.10 2055

Now, I have installed nProbeWin-x64-7.4.160623 (demo version) and ntopng 2.4.160627 - Community Edition which includes nDPI version 20150518 on windows 2008 server (refer snapshot "ntopng Error 3.jpg"). After this, I have started redis service, nProbe and ntopng services from windows service. Opened URL with http://127.0.0.1:3000/ and logged in through 'admin' & 'admin'. Now, I am getting following error in runtime status (refer snapshots "ntopng Error 0.jpg" and "ntopng Error 1.jpg"). Subsequently, I am not getting any layer 7 protocol information (refer snapshot "ntopng Error 2.jpg") from any host as nDPI performs. Please suggest me the area that I have missed in configuring ntopng or nProbe or nDPI to monitor layer 7 application protocols performance.

Let me know if you need any further information on this.

Regards, Surajit ntopng error 0 ntopng error 1 ntopng error 2 ntopng error 3

surajit26 commented 8 years ago

Hi Lucaderi/ simonemainardi,

Any update on this please (refer above attached snapshot).

Thanks in advance.

Regards, Surajit

surajit26 commented 8 years ago

Ok. Thanks a lot for your all support. I am moving to manageengine netflow analyzer for their better customer care support. Cheers, Surajit

lucaderi commented 8 years ago

@surajit26 Feel free to use a non ntopng tool, but don't be impatient as the community edition is support in best effort mode. What you see is correct (beside the log warnings that will be fixed). The reason why DPI is limited (it is not correct to say that you have no DPI at all as ntopng detected netbios for instance), is because your router is not exporting it and thus on the collector side all we can do is guess the DPI protocol but not to compute it. Note that your need to enable in preferences the RRDs for DPI protocols so you can enable charts.

surajit26 commented 8 years ago

Hi Lucaderi,

Thank you for writing to me. I was confused a little. Would suggest to maintain a SLA including escalation matrix to avoid confusion. Can you please suggest how to enable in preferences the RRDs for DPI protocols so I can enable charts. Unless I can view layer 7 application protocol wise bandwidth utilization, I can't go further for up-gradation. Your suggestion will be highly appreciated. Thanks, Surajit

simonemainardi commented 8 years ago

You can enable RRD for DPI from the preferences:

again, as you are exporting netflow data from the routers, you can't leverage a fully-featured nDPI engine, Packets stay on the routers so nProbe/ntopng/nDPI can't inspect them and can just guess the protocols on the basis of port numbers.

If your aim is to have an highly accurate, comprehensive, view of all the protocols used, then you should consider mirroring/tap routers traffic to the nProbes (not just the netflow but the actual packets)

surajit26 commented 8 years ago

Hi Simone, Appreciated your support. It had already been enabled earlier (refer attached snapshot). Can you please suggest the way/ process to mirror/tap routers traffic to the nProbe. Better to have nProbe with license version right? Please confirm. Thanks a ton again for your support. ntopng ndpi rrd

simonemainardi commented 8 years ago

in order to process mirror traffic you should make sure your router has a mirror port. If that is the case, then you can connect that port to an host running nProbe, say on interface <interface name>. At that point is very easy:

nprobe host:   ./nprobe -i <interface name> -n none --zmq tcp://*:5556
ntopng host:   ./ntopng -i tcp://<nprobe host ip>:5556

as nProbe will be able to see the traffic, a full nDPI detection will be performed.

Alternatively you can use a network tap to intercept traffic that is flowing on a wire, say for example from a switch to a router. Then you can connect the tap to a couple of interfaces of an host running nprobe and repeat the steps above.

surajit26 commented 8 years ago

Thanks a lot. Had tried this command also earlier. FYI, I installed both ntopng and nProbe in same server and same drive (c:). This may cause the issue as during installation I was informed redis & other prerequisites are already installed. What you say? I will send you commands output soon for further recommendation. Regards, Surajit

simonemainardi commented 8 years ago

no that is not an issue it is just a notification that redis & other software are already installed. Don't worry about that.

surajit26 commented 8 years ago

Can you take it remotely please. I can share login credentials separately if you want.

surajit26 commented 8 years ago

Can you take it remotely please. I can share login credentials separately if you want.

lucaderi commented 8 years ago

Please send your credentials via email (no github)

lucaderi commented 8 years ago

I have looked at the system and beside some warning everything seems to work as expected. What is the exact problem you want us to look at?

surajit26 commented 8 years ago

It is great pleasure for me.

I am not able to view other protocols (layer 7 protocols) wise traffic that are supported by nDPI since, all applications such as facebook, FTP, youtube, twitter, Skype, VRRP, dropbox, google map are allowed in the network.

surajit26 commented 8 years ago

ntopng error 5 ntopng error 6 I am not able to view this while clicking on any host since all facebook, FTP, youtube, twitter, Skype, VRRP, dropbox, google map are allowed in the network.

lucaderi commented 8 years ago

The 'P' in DPI stands for Packet. As your router sends nProbe flows (not packets) DPI cannot be applied on flows, and thus the protocol detection is limited because of this limitation. You can remove this limitation putting ntopng to listen on a physical packet interface without your router exporting flows.

surajit26 commented 8 years ago

So you are suggesting me to configure it to ethernet switch port?

lucaderi commented 8 years ago

Yes

surajit26 commented 8 years ago

OK. Let's try for good luck. Hope, will be able to revert back with some good news.

surajit26 commented 8 years ago

My badluck..

Attaching output of following command: nprobe host: ./nprobe -i -n none --zmq tcp://*:5556 ntopng host: ./ntopng -i tcp://:5556

Had also configured an ethernet switch port (connected with router) however, same is not showing nDPI application protocols details (e.g. Facebook, Teitter, FTP, youtube, etc). .nprobe -i interface name -n none --zmq tcp5556.txt .ntopng -i tcpnprobe host ip5556.txt ![Uploading NTOPng nDPI RRD Error.jpg…]()

lucaderi commented 8 years ago

You need to enable DPI in the template (-t) adding for instance %L7_PROTO / %L7_PROTO_NAME Example nprobe -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %L7_PROTO %L7_PROTO_NAME" ....

simonemainardi commented 8 years ago

also the interface name is missing after -i in the nprobe command

ntop / ntopng

No Data Available -- Windows -- Netflow #542