search

ntop / nProbe

Open source components and extensions for nProbe
http://ntop.org
GNU General Public License v2.0
1.62k stars 44 forks source link

Netflow Template V9 #51

Closed sdkjain closed 7 years ago

sdkjain commented 8 years ago

Hi,

I have two windows machines(MC1 and MC2), on MC1 I installed latest NtopNG and on MC2 I installed latest Nprobe from ntop.org website and configured MC2 to send Flow to MC1.

Issue # 1: I used default nprobe configuration i.e I didnt changed any config. When I checked the traffic on MC2 I found that the flow exported in Netflow v5 by default. I want to know whether I can change the Netflow template of the nprobe. If yes, then how ? I want to add L7/application level traffic distribution which is not present in default Nprobe Netflow template.

Issue # 2: On MC1 I can see on NtopNG Web UI that Netflow traffic from MC2 is received, but no detail about what is there in that Netflow traffic. How can I see how MC1 is consuming Netflow from MC2 ? Do I need to configure MC1 for it analysing/consuming that Netflow from MC2 ?

Issue # 3: I want to know the application protocol level traffic of MC2 where the probe is installed. How can I do that?

Hope response soon !

Thanks, Sourabh.

ValentinaViscarelli commented 8 years ago

@sdkjain

Issue # 1: yes you can do this. Please use in nprobe "-V 9" and -T option (with -T parameter you have to specify the template). If you run "nprobe /c --help" you can find the parameters that you can use with -T option. For example: -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS %IPV4_SRC_MASK %IPV4_DST_MASK"

For L7/application information you can use these: [NFv9 57590][IPFIX 35632.118] %L7_PROTO Layer 7 protocol (numeric) [NFv9 57591][IPFIX 35632.119] %L7_PROTO_NAME Layer 7 protocol name

Issue # 2: I think the problem is how you run nprobe and ntopng. Do you use zmq between nprobe and ntopng? Please read 3.7 of user's guide: http://www.ntop.org/wp-content/uploads/2013/03/nProbe_UserGuide.pdf

Issue # 3: I'm not sure I understand what you want. Could you please explain me better? Thanks

sdkjain commented 8 years ago

Hi Valentina,

Thanks for your response. I will try the steps for # 1 which u mentioned and check.

2 : My doubt is: When I captured traffic from nprobe MC2 (via wireshark) and found that Netflow format data(Containing 20+ flow records) is sent to my ntopNG MC1. When I checked at UI on MC1, I can only see the Netflow traffic amount, but no information about Flow records in that Netflow data. So I wanted to know how does NtopNG processing this Netflow Data and is ther any way it can show all flow records(20+) at UI ?

3 : From MC2(where nprobe is installed) I want to know traffic based on application layer protocol(HTTP, DNS, FTP, Etc). How does changing the Netflow template to v9 with L7 enabling work, is it based on port no.?

And how does this work when the application data is excrypted ?

Thanks, Sourabh

ValentinaViscarelli commented 8 years ago

2: I think this problem is why the configurations (input parameters) you use to run nprobe and ntopng are wrong. Could you send me the nprobe and ntopng configuration?

3: If you insert %L7_PROTO and %L7_PROTO_NAME in template you will have the application layer protocol, but before you have to resolve point 2. No it isn't based on port number for DPI. We use this: https://github.com/ntop/nDPI

sdkjain commented 8 years ago

Hi,

How do I send the conf. to u ? is there any data file stored on disk which I can send u ?

ValentinaViscarelli commented 8 years ago

How do you run ntopng and nprobe? Maybe are they run on system startup?

sdkjain commented 8 years ago

I just installed on my PC(Windows) the setup files and now they automatically start at startup as they are added as a service in Windows.

sdkjain commented 8 years ago

nprobe

Attached is the file which shows the Netflow recieved from the nprobe machine(10.171.40.89). Whats the traffic in that Netflow is what I cant see in detail. Am I clear now ?

ValentinaViscarelli commented 8 years ago

@sdkjain I understood the problem and this is why the "default service" is not good for your network architecture". First of all you have remove default services. I explain it for nprobe, but you have to do the same for ntopng. 1) open terminal as administrator and go to nprobe installation folder (c:/program files/nprobe) and run "nprobe /r nprobe" 2) If you do "nprobe /h" you can see the option, and they are /c, /r, /i and /h. I suggest you use /c option (manual mode) until everything works, after you can install the service with /i.

You have also to do this for ntopng.

Then open two terminal and run these command (one for each console):

nprobe /c -i 0 -n none --zmq "tcp://*:5556" -V 9 -T "%FIRST_SWITCHED %LAST_SWITCHED %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS %IN_PKTS %IN_BYTES %IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %L4_SRC_PORT %L4_DST_PORT %L7_PROTO %L7_PROTO_NAME"

ntopng /c -i "tcp://10.171.40.89:5556"

Attention: in nprobe "-i 0" is the interface id where you receive the traffic, change it if is different.

sdkjain commented 8 years ago

Hi,

Some hoe its no working. I am running it on two machines. on 10.141.40.89(Nprobe): C:\Program Files\nProbe>nprobe /c -i 0 -n none --zmq "tcp://10.171.41.50:5556" - V 9 -T "%FIRST_SWITCHED %LAST_SWITCHED %FLOW_START_MILLISECONDS %FLOW_END_MILLIS ECONDS %IN_PKTS %IN_BYTES %IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %L4_SRC_PORT %L4_DST_PORT %L7_PROTO %L7_PROTO_NAME" Running nProbe for Windows. I 05/Apr/2016 12:56:31 [nprobe.c:3157] ERROR: Invalid nProbe license (nprobe.licen se) [Missing license file] 05/Apr/2016 12:56:31 [nprobe.c:3164] ERROR: ****

05/Apr/2016 12:56:31 [nprobe.c:3165] ERROR: 05/Apr/2016 12:56:31 [nprobe.c:3166] ERROR: \ Switching to DEMO MODE (missing valid license) 05/Apr/2016 12:56:31 [nprobe.c:3167] ERROR: 05/Apr/2016 12:56:31 [nprobe.c:3168] ERROR: \ Create your nProbe license at 05/Apr/2016 12:56:31 [nprobe.c:3169] ERROR: \ http://www.nmon.net/mklicen se/ 05/Apr/2016 12:56:31 [nprobe.c:3170] ERROR: 05/Apr/2016 12:56:31 [nprobe.c:3171] ERROR: **

05/Apr/2016 12:56:31 [nprobe.c:6639] ERROR: ****

05/Apr/2016 12:56:31 [nprobe.c:6640] ERROR: * NOTE: This is a DEMO version limit ed to 25000 flows export. * 05/Apr/2016 12:56:31 [nprobe.c:6641] ERROR: ****

05/Apr/2016 12:56:31 [nprobe.c:4546] WARNING: The output interfaceId is set to 0 : did you forget to use -Q perhaps ? 05/Apr/2016 12:56:31 [nprobe.c:4549] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 05/Apr/2016 12:56:31 [nprobe.c:4627] Welcome to nProbe Pro v.7.2.151020 ($Revisi on: 4384 $) for Windows 05/Apr/2016 12:56:31 [nprobe.c:4637] Running on Windows 05/Apr/2016 12:56:31 [nprobe.c:4648] [LICENSE] nProbe SystemId: 2365445862-b205a 206 05/Apr/2016 12:56:31 [nprobe.c:6657] Welcome to nProbe v.7.2.151020 for Windows 05/Apr/2016 12:56:31 [nprobe.c:5914] Using NetFlow Packet Payload Len: 1472 05/Apr/2016 12:56:31 [nprobe.c:5970] WARNING: Your template lacks some important fields 05/Apr/2016 12:56:31 [nprobe.c:5971] WARNING: Unless you know what you are doing , make sure 05/Apr/2016 12:56:31 [nprobe.c:5972] WARNING: your template (-T) contains at lea st 05/Apr/2016 12:56:31 [nprobe.c:5973] WARNING: %IPV4_SRC_ADDR %IPV4_DST_ADDR %PRO TOCOL 05/Apr/2016 12:56:31 [nprobe.c:5974] WARNING: %L4_SRC_PORT %L4_DSTPORT 05/Apr/2016 12:56:31 [nprobe.c:6032] WARNING: Protocol will be ignored (your tem plate lacks %PROTOCOL) 05/Apr/2016 12:56:31 [plugin.c:1005] 0 plugin(s) enabled 05/Apr/2016 12:56:31 [nprobe.c:6311] Each flow is 90 bytes long 05/Apr/2016 12:56:31 [nprobe.c:6312] The # packets per flow has been set to 15 05/Apr/2016 12:56:31 [nprobe.c:6331] Non IPv4/v6 traffic is discarded according to the template 05/Apr/2016 12:56:31 [nprobe.c:5199] Using packet capture length 1600 05/Apr/2016 12:56:31 [nprobe.c:6876] Flows ASs will not be computed (missing Geo IP support) 05/Apr/2016 12:56:31 [nprobe.c:6961] Capturing packets from interface \Device\NP F{FE260B3A-E6E8-4385-A1F9-AFF2CF023054} [snaplen: 1600 bytes] 05/Apr/2016 12:56:31 [util.c:4008] ERROR: Unable to bind ZMQ endpoint tcp://10.1 71.41.50:5556: Unknown error 05/Apr/2016 12:56:31 [nprobe.c:7169] nProbe started successfully

sdkjain commented 8 years ago

On 10.171.41.50(NtopNG):

C:\Program Files\ntopng>ntopng /c -i "tcp://127.0.0.1:5556" Starting ntopg Running ntopng. 05/Apr/2016 12:55:59 [Prefs.cpp:820] Logging into C:\Users\soja\Documents\ntopng.log 05/Apr/2016 12:55:59 [Ntop.cpp:980] Setting local networks to 127.0.0.0/8 05/Apr/2016 12:55:59 [Redis.cpp:106] Successfully connected to redis 127.0.0.1:6379@0 05/Apr/2016 12:55:59 [NtopPro.cpp:116] [LICENSE] Read license from Redis [] 05/Apr/2016 12:55:59 [NtopPro.cpp:158] ERROR: [LICENSE] Invalid or missing ntopng License [Empty license file] 05/Apr/2016 12:55:59 [NtopPro.cpp:171] WARNING: [LICENSE] ntopng will now run in pro mode for 10 minutes 05/Apr/2016 12:55:59 [NtopPro.cpp:173] WARNING: [LICENSE] before returning to community mode 05/Apr/2016 12:55:59 [NtopPro.cpp:174] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org 05/Apr/2016 12:55:59 [NtopPro.cpp:175] WARNING: [LICENSE] or run ntopng in community mode starting 05/Apr/2016 12:55:59 [NtopPro.cpp:176] WARNING: [LICENSE] ntopng --community 05/Apr/2016 12:55:59 [Ntop.cpp:1199] Registered interface tcp://127.0.0.1:5556 [id: 2] 05/Apr/2016 12:55:59 [Ntop.cpp:1212] Registered interface view tcp://127.0.0.1:5556 [id: 2] 05/Apr/2016 12:55:59 [HTTPserver.cpp:458] HTTPS Disabled: missing SSL certificate C:\Program Files\ntopng\httpdocs/ssl/ntopng-cert.pem 05/Apr/2016 12:55:59 [HTTPserver.cpp:460] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL. 05/Apr/2016 12:55:59 [HTTPserver.cpp:503] Web server dirs [C:\Program Files\ntopng\httpdocs][C:\Program Files\ntopng\scripts] 05/Apr/2016 12:55:59 [HTTPserver.cpp:506] HTTP server listening on port 3000 05/Apr/2016 12:55:59 [main.cpp:295] Working directory: C:\Users\soja\Documents 05/Apr/2016 12:55:59 [main.cpp:297] Scripts/HTML pages directory: C:\Program Files\ntopng 05/Apr/2016 12:55:59 [Ntop.cpp:258] Welcome to ntopng x64 v.2.3.160306 - (C) 1998-16 ntop.org 05/Apr/2016 12:55:59 [Ntop.cpp:263] Built on Windows 05/Apr/2016 12:55:59 [PeriodicActivities.cpp:53] Started periodic activities loop... 05/Apr/2016 12:55:59 [RuntimePrefs.cpp:34] Dumping alerts into syslog 05/Apr/2016 12:55:59 [NtopPro.cpp:233] [LICENSE] ntopng systemId: 779377967-9205a1d6 05/Apr/2016 12:55:59 [NtopPro.cpp:238] [LICENSE] ntopng is starting in demo mode 05/Apr/2016 12:55:59 [NetworkInterface.cpp:1434] Started packet polling on interface tcp://127.0.0.1:5556 [id: 2]... 05/Apr/2016 12:56:00 [CollectorInterface.cpp:104] Collecting flows on tcp://127.0.0.1:5556 [ntopng->nprobe] 05/Apr/2016 13:06:29 [Lua.cpp:4815] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\pro\dashboard.lua][C:\Program Files\ntopng\scripts\lua\pro\dashboard.lua:1: '=' expected] 05/Apr/2016 13:06:30 [Lua.cpp:4815] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\pro\dashboard.lua][C:\Program Files\ntopng\scripts\lua\pro\dashboard.lua:1: '=' expected] 05/Apr/2016 13:06:56 [main.cpp:37] Shutting down... 05/Apr/2016 13:06:56 [main.cpp:34] Ok I am leaving now

C:\Program Files\ntopng>ntopng /c -i "tcp://10.171.41.50:5556" Starting ntopg Running ntopng. 05/Apr/2016 13:07:14 [Prefs.cpp:820] Logging into C:\Users\soja\Documents\ntopng.log 05/Apr/2016 13:07:14 [Ntop.cpp:980] Setting local networks to 127.0.0.0/8 05/Apr/2016 13:07:14 [Redis.cpp:106] Successfully connected to redis 127.0.0.1:6379@0 05/Apr/2016 13:07:14 [NtopPro.cpp:116] [LICENSE] Read license from Redis [] 05/Apr/2016 13:07:14 [NtopPro.cpp:158] ERROR: [LICENSE] Invalid or missing ntopng License [Empty license file] 05/Apr/2016 13:07:14 [NtopPro.cpp:171] WARNING: [LICENSE] ntopng will now run in pro mode for 10 minutes 05/Apr/2016 13:07:14 [NtopPro.cpp:173] WARNING: [LICENSE] before returning to community mode 05/Apr/2016 13:07:14 [NtopPro.cpp:174] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org 05/Apr/2016 13:07:14 [NtopPro.cpp:175] WARNING: [LICENSE] or run ntopng in community mode starting 05/Apr/2016 13:07:14 [NtopPro.cpp:176] WARNING: [LICENSE] ntopng --community 05/Apr/2016 13:07:14 [Ntop.cpp:1199] Registered interface tcp://10.171.41.50:5556 [id: 3] 05/Apr/2016 13:07:14 [Ntop.cpp:1212] Registered interface view tcp://10.171.41.50:5556 [id: 3] 05/Apr/2016 13:07:14 [HTTPserver.cpp:458] HTTPS Disabled: missing SSL certificate C:\Program Files\ntopng\httpdocs/ssl/ntopng-cert.pem 05/Apr/2016 13:07:14 [HTTPserver.cpp:460] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL. 05/Apr/2016 13:07:14 [HTTPserver.cpp:503] Web server dirs [C:\Program Files\ntopng\httpdocs][C:\Program Files\ntopng\scripts] 05/Apr/2016 13:07:14 [HTTPserver.cpp:506] HTTP server listening on port 3000 05/Apr/2016 13:07:14 [main.cpp:295] Working directory: C:\Users\soja\Documents 05/Apr/2016 13:07:14 [main.cpp:297] Scripts/HTML pages directory: C:\Program Files\ntopng 05/Apr/2016 13:07:14 [Ntop.cpp:258] Welcome to ntopng x64 v.2.3.160306 - (C) 1998-16 ntop.org 05/Apr/2016 13:07:14 [Ntop.cpp:263] Built on Windows 05/Apr/2016 13:07:14 [PeriodicActivities.cpp:53] Started periodic activities loop... 05/Apr/2016 13:07:14 [RuntimePrefs.cpp:34] Dumping alerts into syslog 05/Apr/2016 13:07:14 [NtopPro.cpp:233] [LICENSE] ntopng systemId: 779377967-9205a1d6 05/Apr/2016 13:07:14 [NtopPro.cpp:238] [LICENSE] ntopng is starting in demo mode 05/Apr/2016 13:07:14 [NetworkInterface.cpp:1434] Started packet polling on interface tcp://10.171.41.50:5556 [id: 3]... 05/Apr/2016 13:07:14 [CollectorInterface.cpp:104] Collecting flows on tcp://10.171.41.50:5556 [ntopng->nprobe]

sdkjain commented 8 years ago

I cant see any flow traffic neither on Nprobe machine nor on NtopNG UI

ValentinaViscarelli commented 8 years ago

It's wrong. You have to run nprobe command on nprobe machine and ntopng on ntopng machine. Why do you have change * with ip_address in zmq parameters on nprobe command? Please use these commands (nprobe for 10.171.40.89 machine and ntopng for 10.171.41.50 machine). Please use exactly these commands (the only thing you need to change is the interface identifier if "0" is not your interface id in -i parameter on nprobe command):

nprobe /c -i 0 -n none --zmq "tcp://*:5556" -V 9 -T "%FIRST_SWITCHED %LAST_SWITCHED %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS %IN_PKTS %IN_BYTES %IPV4_SRC_ADDR %IPV4_DST_ADDR %PROTOCOL %INPUT_SNMP %OUTPUT_SNMP %L4_SRC_PORT %L4_DST_PORT %L7_PROTO %L7_PROTO_NAME"

ntopng /c -i "tcp://10.171.40.89:5556"

Make sure the nprobe command is a single line on the console.

sdkjain commented 8 years ago

I did that but still no flows, I can see some errors on console on Ntopng machine: 05/Apr/2016 15:24:53 [CollectorInterface.cpp:104] Collecting flows on tcp://10.171.40.89:5556 [ntopng->nprobe] 05/Apr/2016 15:24:59 [Lua.cpp:2083] ERROR: Error 'fetching cdp from rra' while calling rrd_fetch_r(C:\Users\soja\Documents\1\rrd\bytes.rrd, AVERAGE): is the RRD corrupted perhaps? 05/Apr/2016 15:25:29 [Lua.cpp:2083] ERROR: Error 'fetching cdp from rra' while calling rrd_fetch_r(C:\Users\soja\Documents\1\rrd\bytes.rrd, AVERAGE): is the RRD corrupted perhaps? 05/Apr/2016 15:25:32 [Lua.cpp:2083] ERROR: Error 'fetching cdp from rra' while calling rrd_fetch_r(C:\Users\soja\Documents\1\rrd\bytes.rrd, AVERAGE): is the RRD corrupted perhaps?

ValentinaViscarelli commented 8 years ago

Please try to remove this folder: C:\Users\soja\Documents\1

sdkjain commented 8 years ago

same error still. Even after deleting the folder '1':

07/Apr/2016 12:06:03 [CollectorInterface.cpp:104] Collecting flows on tcp://10.171.40.89:5556 [ntopng->nprobe] 07/Apr/2016 12:06:07 [Lua.cpp:2083] ERROR: Error 'fetching cdp from rra' while calling rrd_fetch_r(C:\Users\soja\Documents\1\rrd\bytes.rrd, AVERAGE): is the RRD corrupted perhaps? 07/Apr/2016 12:06:09 [Lua.cpp:2083] ERROR: Error 'fetching cdp from rra' while calling rrd_fetch_r(C:\Users\soja\Documents\1\rrd\bytes.rrd, AVERAGE): is the RRD corrupted perhaps? 07/Apr/2016 12:06:11 [Lua.cpp:2083] ERROR: Error 'fetching cdp from rra' while calling rrd_fetch_r(C:\Users\soja\Documents\1\rrd\bytes.rrd, AVERAGE): is the RRD corrupted perhaps?

sdkjain commented 8 years ago

please reply !

ValentinaViscarelli commented 8 years ago

For ntopng problem do this:

For nprobe:

What windows version are you using?

sdkjain commented 8 years ago

Hi,

I can see nprobe packet export. But at ntopNG nothing on UI.

Nprobe: 11/Apr/2016 14:53:41 [engine.c:2496] Emitting Flow: [->][udp] FE00::3105:1C08:8A A7:60A2:546 -> FF02::1:2:547 [1 pkt/139 bytes][ifIdx 65535->65535][0.0 sec][DHCP V6/103][init Unknown][AS: 0 -> 0]

NtopNG: temp

ValentinaViscarelli commented 8 years ago

Please could you post startup phase log of ntopng and nprobe (included the command you use)?

Thanks

sdkjain commented 8 years ago

I have turned of RRD from pref.: C:\Program Files\ntopng>ntopng /c -i "tcp://10.171.40.89:5556" Starting ntopg Running ntopng. 11/Apr/2016 14:51:04 [Prefs.cpp:820] Logging into C:\Users\soja\Documents\ntopng.log 11/Apr/2016 14:51:04 [Ntop.cpp:980] Setting local networks to 127.0.0.0/8 11/Apr/2016 14:51:04 [Redis.cpp:106] Successfully connected to redis 127.0.0.1:6379@0 11/Apr/2016 14:51:04 [NtopPro.cpp:116] [LICENSE] Read license from Redis [] 11/Apr/2016 14:51:04 [NtopPro.cpp:158] ERROR: [LICENSE] Invalid or missing ntopng License [Empty license file] 11/Apr/2016 14:51:04 [NtopPro.cpp:171] WARNING: [LICENSE] ntopng will now run in pro mode for 10 minutes 11/Apr/2016 14:51:04 [NtopPro.cpp:173] WARNING: [LICENSE] before returning to community mode 11/Apr/2016 14:51:04 [NtopPro.cpp:174] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org 11/Apr/2016 14:51:04 [NtopPro.cpp:175] WARNING: [LICENSE] or run ntopng in community mode starting 11/Apr/2016 14:51:04 [NtopPro.cpp:176] WARNING: [LICENSE] ntopng --community 11/Apr/2016 14:51:05 [Ntop.cpp:1199] Registered interface tcp://10.171.40.89:5556 [id: 1] 11/Apr/2016 14:51:05 [Ntop.cpp:1212] Registered interface view tcp://10.171.40.89:5556 [id: 1] 11/Apr/2016 14:51:05 [HTTPserver.cpp:458] HTTPS Disabled: missing SSL certificate C:\Program Files\ntopng\httpdocs/ssl/ntopng-cert.pem 11/Apr/2016 14:51:05 [HTTPserver.cpp:460] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL. 11/Apr/2016 14:51:05 [HTTPserver.cpp:503] Web server dirs [C:\Program Files\ntopng\httpdocs][C:\Program Files\ntopng\scripts] 11/Apr/2016 14:51:05 [HTTPserver.cpp:506] HTTP server listening on port 3000 11/Apr/2016 14:51:05 [main.cpp:295] Working directory: C:\Users\soja\Documents 11/Apr/2016 14:51:05 [main.cpp:297] Scripts/HTML pages directory: C:\Program Files\ntopng 11/Apr/2016 14:51:05 [Ntop.cpp:258] Welcome to ntopng x64 v.2.3.160306 - (C) 1998-16 ntop.org 11/Apr/2016 14:51:05 [Ntop.cpp:263] Built on Windows 11/Apr/2016 14:51:05 [PeriodicActivities.cpp:53] Started periodic activities loop... 11/Apr/2016 14:51:05 [RuntimePrefs.cpp:34] Dumping alerts into syslog 11/Apr/2016 14:51:05 [NtopPro.cpp:233] [LICENSE] ntopng systemId: 779377967-9205a1d6 11/Apr/2016 14:51:05 [NtopPro.cpp:238] [LICENSE] ntopng is starting in demo mode 11/Apr/2016 14:51:05 [NetworkInterface.cpp:1434] Started packet polling on interface tcp://10.171.40.89:5556 [id: 1]... 11/Apr/2016 14:51:06 [CollectorInterface.cpp:104] Collecting flows on tcp://10.171.40.89:5556 [ntopng->nprobe] 11/Apr/2016 14:52:42 [Lua.cpp:2083] ERROR: Error 'fetching cdp from rra' while calling rrd_fetch_r(C:\Users\soja\Documents\1\rrd\bytes.rrd, AVERAGE): is the RRD corrupted perhaps?

sdkjain commented 8 years ago

FYI, I am using Windows 10

sdkjain commented 8 years ago

I had one more doubt below:

Can I use NtopNG as a Netflow collector, say which is directly exported from Routers ??

ValentinaViscarelli commented 8 years ago

Sorry, but you never said before you have a router that exports netflow data!! The configuration I said you is if you have mirror traffic on nprobe machine interface. If your router exports netflow data you can see this traffic (with another configuration than I sent you) on ntopng but nprobe will not able to understand L7 protocol (nprobe doesn't have enough information). Your router is able to understand and export L7 protocol? If you have mirror traffic nprobe is able to understand and export L7 protocol.

sdkjain commented 8 years ago

No, I dont have a router. I am having two machine setup as you understood. I just asked you if it can be used or not.

Ignore that now, first lets focus on old issue :)

ValentinaViscarelli commented 8 years ago

Ok sorry. I didn't understand. It's strange doesn't work, but let try some workarounds. Please try to move ntopng in the same machine as nprobe (so use just one machine) and run nprobe and ntopng like this:

nprobe /c -i 0 -n none --zmq "tcp://127.0.0.1:5556" -V 9 -T "%FIRST_SWITCHED %LAST_SWITCHED %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS %IN_PKTS %IN_BYTES %IPV4_SRC_ADDR %IPV4_DST_ADDR %PROTOCOL %INPUT_SNMP %OUTPUT_SNMP %L4_SRC_PORT %L4_DST_PORT %L7_PROTO %L7_PROTO_NAME"

ntopng /c -i "tcp://127.0.0.1:5556"

Remeber use in nprobe your interface (-i parameter)

BTW Is there any reason why you use two machine?

sdkjain commented 8 years ago

Hi,

Tried on same machine also. I can see any flow at ntopNG UI :(

simonemainardi commented 8 years ago

The main point here is that you are on windows10. I think the first think to do is to make sure nProbe correctly captures traffic and generates flows.

Run nprobe and then stop it. Upon exit it should print some statistics. Check and see if flow export stats are non-zero. Following is an example output:

12/Apr/2016 09:50:08 [nprobe.c:430] Received shutdown request... [signal: 2]
12/Apr/2016 09:50:09 [engine.c:2641] About to flush hash (threadId 0)
12/Apr/2016 09:50:09 [engine.c:2643] Completed hash walk (thread 0)
12/Apr/2016 09:50:12 [cache.c:1224] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec]
12/Apr/2016 09:50:12 [nprobe.c:2603] Processed packets: 5 (max bucket search: 0)
12/Apr/2016 09:50:12 [nprobe.c:2586] Fragment queue length: 0
12/Apr/2016 09:50:12 [nprobe.c:2612] Flow export stats: [876 bytes/3 pkts][2 flows/1 pkts sent]
12/Apr/2016 09:50:12 [nprobe.c:2622] Flow drop stats:   [0 bytes/0 pkts][0 flows]
12/Apr/2016 09:50:12 [nprobe.c:2627] Total flow stats:  [876 bytes/3 pkts][2 flows/1 pkts sent]
sdkjain commented 8 years ago

Hi,

Below are the log prints at exit:

12/Apr/2016 14:14:45 [nprobe.c:429] Received shutdown request... [signal: 2] 12/Apr/2016 14:14:45 [nprobe.c:5989] fetchPcapPackets(threadId=0) terminated 12/Apr/2016 14:14:45 [nprobe.c:4949] nProbe is shutting down... 12/Apr/2016 14:14:45 [nprobe.c:4985] Exporting pending buckets... 12/Apr/2016 14:14:45 [engine.c:2636] About to flush hash (threadId 0) 12/Apr/2016 14:14:45 [nprobe.c:365] Packet stats (IN/OUT): 243/0 pkts rcvd/dropped [0.0%] [Last 87/0 pkts rcvd/dropped] 12/Apr/2016 14:14:45 [engine.c:2638] Completed hash walk (thread 0) 12/Apr/2016 14:14:45 [engine.c:2504] Emitting Flow: [->][udp] 10.171.41.47:55208 -> 224.0.0.252:5355 [1 pkt/50 bytes][ifIdx 65535->65535][0.0 sec][LLMNR/154][init Unknown][AS: 0 -> 0] . . . . -> 0] 12/Apr/2016 14:14:46 [engine.c:2504] Emitting Flow: [->][unknown] 3FFE::31:0 -> FF02::1:FF00:220:0 [1 pkt/72 bytes][ifIdx 65535->65535][0.0 sec][NEIGHBOR SOLIC][ICMPV6/102][init Unknown][AS: 0 -> 0] 12/Apr/2016 14:14:46 [engine.c:2504] Emitting Flow: [->][udp] 10.86.96.194:63303 -> 239.255.255.250:1900 [1 pkt/129 bytes][ifIdx 65535->65535][0.0 sec][SSDP/12][init Unknown][AS: 0 -> 0] 12/Apr/2016 14:14:46 [engine.c:2504] Emitting Flow: [->][udp] 10.171.143.34:55847 -> 239.255.255.250:1900 [1 pkt/165 bytes][ifIdx 65535->65535][0.0 sec][SSDP/12][init Unknown][AS: 0 -> 0] 12/Apr/2016 14:14:47 [nprobe.c:5006] Pending buckets have been exported... 12/Apr/2016 14:14:49 [engine.c:3187] Export thread terminated [exportQueue=0] 12/Apr/2016 14:14:49 [nprobe.c:5065] Flushing queued flows... 12/Apr/2016 14:14:49 [nprobe.c:5068] Freeing memory... 12/Apr/2016 14:14:49 [nprobe.c:365] Packet stats (IN/OUT): 393/0 pkts rcvd/dropped [0.0%] [Last 150/0 pkts rcvd/dropped]

ValentinaViscarelli commented 8 years ago

Do you have a big amount of traffic? If no you have use just ntopng. Turn off nprobe and run ntopng like this:

ntopng /c -i 0

change -i 0 in such way that works for your interface.

sdkjain commented 8 years ago

Got below error: C:\Program Files\ntopng>ntopng /c -i 0 Starting ntopg Running ntopng. 12/Apr/2016 15:46:35 [Prefs.cpp:820] Logging into C:\Users\soja\Documents\ntopng.log 12/Apr/2016 15:46:35 [Ntop.cpp:980] Setting local networks to 127.0.0.0/8 12/Apr/2016 15:46:35 [Redis.cpp:106] Successfully connected to redis 127.0.0.1:6379@0 12/Apr/2016 15:46:35 [NtopPro.cpp:116] [LICENSE] Read license from Redis [] 12/Apr/2016 15:46:35 [NtopPro.cpp:158] ERROR: [LICENSE] Invalid or missing ntopng License [Empty license file] 12/Apr/2016 15:46:35 [NtopPro.cpp:171] WARNING: [LICENSE] ntopng will now run in pro mode for 10 minutes 12/Apr/2016 15:46:35 [NtopPro.cpp:173] WARNING: [LICENSE] before returning to community mode 12/Apr/2016 15:46:35 [NtopPro.cpp:174] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org 12/Apr/2016 15:46:35 [NtopPro.cpp:175] WARNING: [LICENSE] or run ntopng in community mode starting 12/Apr/2016 15:46:35 [NtopPro.cpp:176] WARNING: [LICENSE] ntopng --community 12/Apr/2016 15:46:35 [NetworkInterface.cpp:134] WARNING: Unable to locate interface Id 0

sdkjain commented 8 years ago

With -i 1: C:\Program Files\ntopng>ntopng /c -i 1 Starting ntopg Running ntopng. 12/Apr/2016 15:47:54 [Prefs.cpp:820] Logging into C:\Users\soja\Documents\ntopng.log 12/Apr/2016 15:47:54 [Ntop.cpp:980] Setting local networks to 127.0.0.0/8 12/Apr/2016 15:47:54 [Redis.cpp:106] Successfully connected to redis 127.0.0.1:6379@0 12/Apr/2016 15:47:54 [NtopPro.cpp:116] [LICENSE] Read license from Redis [] 12/Apr/2016 15:47:54 [NtopPro.cpp:158] ERROR: [LICENSE] Invalid or missing ntopng License [Empty license file] 12/Apr/2016 15:47:54 [NtopPro.cpp:171] WARNING: [LICENSE] ntopng will now run in pro mode for 10 minutes 12/Apr/2016 15:47:54 [NtopPro.cpp:173] WARNING: [LICENSE] before returning to community mode 12/Apr/2016 15:47:54 [NtopPro.cpp:174] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org 12/Apr/2016 15:47:54 [NtopPro.cpp:175] WARNING: [LICENSE] or run ntopng in community mode starting 12/Apr/2016 15:47:54 [NtopPro.cpp:176] WARNING: [LICENSE] ntopng --community 12/Apr/2016 15:47:54 [PcapInterface.cpp:85] Reading packets from interface NPF{A163B00A-7137-4DF1-B0E8-9B86E75EE645}... 12/Apr/2016 15:47:54 [PcapInterface.cpp:90] WARNING: Unable to set packet capture direction 12/Apr/2016 15:47:54 [Ntop.cpp:1199] Registered interface NPF{A163B00A-7137-4DF1-B0E8-9B86E75EE645} [id: 0] 12/Apr/2016 15:47:54 [Ntop.cpp:1001] ERROR: Unable to find interface Id 1 12/Apr/2016 15:47:54 [NetworkInterfaceView.cpp:35] ERROR: Unknown interface 1: skept 12/Apr/2016 15:47:54 [HTTPserver.cpp:458] HTTPS Disabled: missing SSL certificate C:\Program Files\ntopng\httpdocs/ssl/ntopng-cert.pem 12/Apr/2016 15:47:54 [HTTPserver.cpp:460] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL. 12/Apr/2016 15:47:54 [HTTPserver.cpp:503] Web server dirs [C:\Program Files\ntopng\httpdocs][C:\Program Files\ntopng\scripts] 12/Apr/2016 15:47:54 [HTTPserver.cpp:506] HTTP server listening on port 3000 12/Apr/2016 15:47:54 [main.cpp:295] Working directory: C:\Users\soja\Documents 12/Apr/2016 15:47:54 [main.cpp:297] Scripts/HTML pages directory: C:\Program Files\ntopng 12/Apr/2016 15:47:54 [Ntop.cpp:258] Welcome to ntopng x64 v.2.3.160306 - (C) 1998-16 ntop.org 12/Apr/2016 15:47:54 [Ntop.cpp:263] Built on Windows 12/Apr/2016 15:47:54 [PeriodicActivities.cpp:53] Started periodic activities loop... 12/Apr/2016 15:47:54 [RuntimePrefs.cpp:34] Dumping alerts into syslog 12/Apr/2016 15:47:54 [NtopPro.cpp:233] [LICENSE] ntopng systemId: 779377967-9205a1d6 12/Apr/2016 15:47:54 [NtopPro.cpp:238] [LICENSE] ntopng is starting in demo mode 12/Apr/2016 15:47:54 [Ntop.cpp:437] Adding 10.171.40.0/23 as local address for NPF{A163B00A-7137-4DF1-B0E8-9B86E75EE645} 12/Apr/2016 15:47:54 [Ntop.cpp:442] Adding 10.171.41.50/32 as IPv4 interface address for NPF{A163B00A-7137-4DF1-B0E8-9B86E75EE645} 12/Apr/2016 15:47:54 [NetworkInterface.cpp:1434] Started packet polling on interface NPF_{A163B00A-7137-4DF1-B0E8-9B86E75EE645} [id: 0]... 12/Apr/2016 15:47:54 [Lua.cpp:4815] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\get_hosts_data.lua][C:\Program Files\ntopng/scripts/lua/modules/lua_utils.lua:92: bad argument #1 to 'pairs' (table expected, got userdata)] 12/Apr/2016 15:47:55 [Lua.cpp:4815] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\get_hosts_data.lua][C:\Program Files\ntopng/scripts/lua/modules/lua_utils.lua:92: bad argument #1 to 'pairs' (table expected, got userdata)] 12/Apr/2016 15:47:55 [Lua.cpp:4815] WARNING: Script failure [C:\Program Files\ntopng\scripts\lua\get_http_hosts_data.lua][C:\Program Files\ntopng\scripts\lua\get_http_hosts_data.lua:76: bad argument #1 to 'pairs' (table expected, got

ValentinaViscarelli commented 8 years ago

Are you sure interface 1 is correct?

you have: 12/Apr/2016 15:47:54 [Ntop.cpp:1001] ERROR: Unable to find interface Id 1 12/Apr/2016 15:47:54 [NetworkInterfaceView.cpp:35] ERROR: Unknown interface 1: skept

If you run "ntopng /c -h", it should say the available interfaces.

sdkjain commented 8 years ago

yes : Available interfaces (-i ):

  1. Intel(R) 82579LM Gigabit Network Connection
simonemainardi commented 8 years ago

Uninstall the current winpcap and then try to install http://www.win10pcap.org/

sdkjain commented 8 years ago

I installled winpcap10 on both machines and uninstalled the old winpcap, still I cant see anything at ntopngUI.

UI is same at what screenshot I shared earlier.

hsluoyz commented 8 years ago

@sdkjain,

Please try Npcap. Npcap is a fork of WinPcap updated for NDIS 6 and many new features. It's sponsored by Nmap project.

The latest version is Npcap 0.07, please try the installer at:

https://github.com/nmap/npcap/releases

ValentinaViscarelli commented 7 years ago

Old task. Please try with the current packages and if you still have the problem, please open a new issue.

Thanks