search

ntop / nProbe

Open source components and extensions for nProbe
http://ntop.org
GNU General Public License v2.0
1.62k stars 44 forks source link

nProbe not capturing flows in proxy mode #10

Closed smerkal closed 8 years ago

smerkal commented 8 years ago

Flow data from a Juniper MX router is not being captured by nProbe running in Proxy mode. Nothing is passed to nTopng via ZMQ.

.pcap file of flow data hitting the nProbe server during the following nProbe run:

tcpdump -n -l -i eth4 -w cflow.pcap port 2055

(rename to cflow.pcap. Github keeps rejecting .zip files) cflow.txt

root@uncsnbox:/etc/nprobe# nprobe -3 2055 -zmq=tcp://*:5556 -n none -b 2 04/Dec/2015 17:40:32 [nprobe.c:3130] Valid nProbe Pro license found 04/Dec/2015 17:40:32 [plugin.c:166] No plugins found in ./plugins 04/Dec/2015 17:40:32 [plugin.c:174] Loading 22 plugins [.so] from /usr/local/lib/nprobe/plugins 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin DHCP Protocol: missing license [/etc/nprobe.license.dhcp] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin Diameter Protocol: missing license [/etc/nprobe.license.diameter] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin DNS Protocol: missing license [/etc/nprobe.license.dns] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin Export Plugin: missing license [/etc/nprobe.license.export] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin FTP Protocol: missing license [/etc/nprobe.license.ftp] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin GTPv0 Signaling Protocol: missing license [/etc/nprobe.license.gtpv0] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin GTPv1 Signaling Protocol: missing license [/etc/nprobe.license.gtpv1] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin GTPv2 Signaling Protocol: missing license [/etc/nprobe.license.gtpv2] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin HTTP Protocol: missing license [/etc/nprobe.license.http] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin IMAP Protocol: missing license [/etc/nprobe.license.email] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin Netflow-Lite Plugin: missing license [/etc/nprobe.license.nflite] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin Oracle Protocol: missing license [/etc/nprobe.license.oracle] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin POP3 Protocol: missing license [/etc/nprobe.license.email] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin System process information: missing license [/etc/nprobe.license.process] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin Radius Protocol: missing license [/etc/nprobe.license.radius] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin RTP Plugin: missing license [/etc/nprobe.license.voippro] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin S1AP Protocol: missing license [/etc/nprobe.license.S1AP] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin SIP Plugin: missing license [/etc/nprobe.license.voippro] 04/Dec/2015 17:40:32 [plugin.c:742] Unable to enable plugin SMTP Protocol: missing license [/etc/nprobe.license.email] 04/Dec/2015 17:40:32 [nprobe.c:4488] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 04/Dec/2015 17:40:32 [nprobe.c:4491] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 04/Dec/2015 17:40:32 [nprobe.c:4552] Welcome to nProbe Pro v.7.2.151202 ($Revision: 4471 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration 04/Dec/2015 17:40:32 [nprobe.c:4562] Running on Ubuntu 14.04.2 LTS 04/Dec/2015 17:40:32 [nprobe.c:4573] [LICENSE] nProbe SystemId: FA623D157104A1D2 04/Dec/2015 17:40:32 [nprobe.c:4620] Tracing enabled 04/Dec/2015 17:40:32 [bgpPlugin.c:375] BGP plugin is disabled (--bgp-port has not been specified) 04/Dec/2015 17:40:32 [dbPlugin.c:49] Initializing DB plugin 04/Dec/2015 17:40:32 [mysqlPlugin.c:111] Initialized MySQL plugin 04/Dec/2015 17:40:32 [plugin.c:248] 3 plugin(s) loaded [3 delete][2 packet]. 04/Dec/2015 17:40:32 [nprobe.c:6526] Welcome to nprobe v.7.2.151202 for x86_64-unknown-linux-gnu 04/Dec/2015 17:40:32 [nprobe.c:5752] Compiling flow templates... 04/Dec/2015 17:40:32 [plugin.c:851] Scanning plugin BGP Update Listener [bgp] 04/Dec/2015 17:40:32 [plugin.c:851] Scanning plugin MySQL DB [db] 04/Dec/2015 17:40:32 [plugin.c:851] Scanning plugin MySQL Plugin [mysql] 04/Dec/2015 17:40:32 [plugin.c:1000] 0 plugin(s) enabled 04/Dec/2015 17:40:32 [nprobe.c:6203] Non IPv4/v6 traffic is discarded according to the template 04/Dec/2015 17:40:32 [util.c:287] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat 04/Dec/2015 17:40:32 [util.c:296] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat 04/Dec/2015 17:40:32 [nprobe.c:6698] IPv6 traffic will NOT be exported/accounted by this probe 04/Dec/2015 17:40:32 [nprobe.c:6699] due to configuration options (e.g. use NetFlow v9) 04/Dec/2015 17:40:32 [nprobe.c:6702] The flows hash has 131072 buckets 04/Dec/2015 17:40:32 [nprobe.c:6704] Flows older than 120 seconds will be exported 04/Dec/2015 17:40:32 [nprobe.c:6707] Flows inactive for at least 30 seconds will be exported 04/Dec/2015 17:40:32 [nprobe.c:6710] Expired flows will not be queued for more than 30 seconds 04/Dec/2015 17:40:32 [nprobe.c:6717] Exported flows with engineType 0 and engineId 112 04/Dec/2015 17:40:32 [nprobe.c:6739] TCP TOS will be ignored and set to 0. 04/Dec/2015 17:40:32 [nprobe.c:6757] After 1 flow packets are sent, we'll delay at least 1 ms 04/Dec/2015 17:40:32 [nprobe.c:6777] Flows will be emitted in NetFlow 5 format 04/Dec/2015 17:40:32 [nprobe.c:6807] Flow input interface index is set to 0 04/Dec/2015 17:40:32 [nprobe.c:6813] Flow output interface index is set to 0 04/Dec/2015 17:40:32 [util.c:2892] WARNING: Don't dropping privileges (required by NetFilter) 04/Dec/2015 17:40:32 [plugin.c:813] Disabling plugin BGP Update Listener (no template is using it) 04/Dec/2015 17:40:32 [plugin.c:813] Disabling plugin MySQL DB (no template is using it) 04/Dec/2015 17:40:32 [plugin.c:813] Disabling plugin MySQL Plugin (no template is using it) 04/Dec/2015 17:40:32 [collect.c:86] Created UDP sockets 04/Dec/2015 17:40:32 [collect.c:90] Created a SCTP socket (53) 04/Dec/2015 17:40:32 [collect.c:145] Flow collector listening on port 2055 (IPv4/v6) 04/Dec/2015 17:40:32 [nprobe.c:6947] Starting 1 packet fetch thread(s) 04/Dec/2015 17:40:32 [nprobe.c:7035] nProbe started successfully 04/Dec/2015 17:40:32 [engine.c:3210] Starting bucket dequeue thread

04/Dec/2015 17:43:25 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec] 04/Dec/2015 17:43:25 [nprobe.c:386] Received shutdown request... [signal: 2] 04/Dec/2015 17:43:26 [nprobe.c:4716] nProbe is shutting down... 04/Dec/2015 17:43:26 [nprobe.c:4752] Exporting pending buckets... 04/Dec/2015 17:43:26 [nprobe.c:4773] Pending buckets have been exported... 04/Dec/2015 17:43:28 [engine.c:3293] Export thread terminated [exportQueue=0] 04/Dec/2015 17:43:28 [nprobe.c:4839] Flushing queued flows... 04/Dec/2015 17:43:28 [nprobe.c:4842] Freeing memory... 04/Dec/2015 17:43:28 [plugin.c:277] Terminating plugins. 04/Dec/2015 17:43:28 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec] 04/Dec/2015 17:43:28 [nprobe.c:4934] Still allocated 0 hash buckets 04/Dec/2015 17:43:28 [nprobe.c:2457] Processed packets: 0 (max bucket search: 0) 04/Dec/2015 17:43:28 [nprobe.c:2440] Fragment queue length: 0 04/Dec/2015 17:43:28 [nprobe.c:2466] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent] 04/Dec/2015 17:43:28 [nprobe.c:2473] Flow collection: [collected pkts: 0][processed flows: 0] 04/Dec/2015 17:43:28 [nprobe.c:2476] Flow drop stats: [0 bytes/0 pkts][0 flows] 04/Dec/2015 17:43:28 [nprobe.c:2481] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent] 04/Dec/2015 17:43:28 [nprobe.c:4947] Cleaning globals 04/Dec/2015 17:43:28 [nprobe.c:4967] nProbe terminated.

lmangani commented 8 years ago

I believe the issue is with the use of -n none. From the command description:

If you specify none as value, no flow will be exported; in this case the -P parameter is mandatory.

Adding a dummy -n collector seems to confirm this is the case:

07/Dec/2015 21:38:15 [nprobe.c:2503] Processed packets: 3356 (max bucket search: 0)
07/Dec/2015 21:38:15 [nprobe.c:2486] Fragment queue length: 0
07/Dec/2015 21:38:15 [nprobe.c:2512] Flow export stats: [1407752 bytes/3356 pkts][2 flows/1 pkts sent]
07/Dec/2015 21:38:15 [nprobe.c:2519] Flow collection: [collected pkts: 0][processed flows: 0]
07/Dec/2015 21:38:15 [nprobe.c:2522] Flow drop stats:   [0 bytes/0 pkts][0 flows]
07/Dec/2015 21:38:15 [nprobe.c:2527] Total flow stats:  [1407752 bytes/3356 pkts][2 flows/1 pkts sent]
smerkal commented 8 years ago

Flows should still be emitted via the zmq endpoint, just not exported to an external collector. The use of the '-n none' parameter keeps a waterfall condition from occurring due to the default behavior of exporting on port 2055 if -n is not specified, which is the same port we are collecting on. This exact scenario worked in the past.

Trying to process the attached pcap file using the '-i ' option just results in collecting and emitting/exporting flow data from ABOUT the packets between the router and the nProbe server, not proxying the flow data contained WITHIN the packets received from the router.

Adding the -i option to the command line in my original post also results in nProbe collecting and emitting flow data via zmq from the sessions between the router and the server as well, no -P option is specified. Flows are emitted via zmq, but not exported as seen below:

nprobe -3 2055 -zmq=tcp://*:5556 -n none -b 2 -i eth4

07/Dec/2015 18:36:07 [engine.c:3210] Starting bucket dequeue thread 07/Dec/2015 18:36:08 [engine.c:2361] New Flow: [udp] 172.31.0.0:50101 -> 172.31.200.50:2055 [00:00:00:00:00:00 -> 00:00:00:00:00:00][vlan 0][tos 0][ifIdx: 65535 -> 65535][subflowId: 0/0x0000][idx=76663] 07/Dec/2015 18:36:08 [pro/pf_ring.c:94] PF_RING stats (Average): 4/0 [0.0 %] pkts rcvd/dropped 07/Dec/2015 18:36:08 [engine.c:2361] New Flow: [udp] 172.31.0.8:50101 -> 172.31.200.50:2056 [00:00:00:00:00:00 -> 00:00:00:00:00:00][vlan 0][tos 0][ifIdx: 65535 -> 65535][subflowId: 0/0x0000][idx=76674] 07/Dec/2015 18:36:08 [engine.c:2361] New Flow: [udp] 172.31.0.8:50103 -> 172.31.200.50:2056 [00:00:00:00:00:00 -> 00:00:00:00:00:00][vlan 0][tos 0][ifIdx: 65535 -> 65535][subflowId: 0/0x0000][idx=76680] 07/Dec/2015 18:36:08 [engine.c:2361] New Flow: [udp] 172.31.0.0:50103 -> 172.31.200.50:2055 [00:00:00:00:00:00 -> 00:00:00:00:00:00][vlan 0][tos 0][ifIdx: 65535 -> 65535][subflowId: 0/0x0000][idx=76669] ^C07/Dec/2015 18:36:09 [pro/pf_ring.c:94] PF_RING stats (Average): 12/0 [0.0 %] pkts rcvd/dropped 07/Dec/2015 18:36:09 [pro/pf_ring.c:105] PF_RING stats (Current): 8/0 [0.0 %] pkts rcvd/dropped 07/Dec/2015 18:36:09 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec] 07/Dec/2015 18:36:09 [nprobe.c:386] Received shutdown request... [signal: 2] 07/Dec/2015 18:36:09 [pro/pf_ring.c:300] Terminated PF_RING packet processing 07/Dec/2015 18:36:10 [nprobe.c:4716] nProbe is shutting down... 07/Dec/2015 18:36:10 [nprobe.c:4722] Waiting for PF_RING termination 07/Dec/2015 18:36:10 [nprobe.c:4731] PF_RING terminated 07/Dec/2015 18:36:10 [nprobe.c:4752] Exporting pending buckets... 07/Dec/2015 18:36:10 [engine.c:2673] About to flush hash (threadId 0) 07/Dec/2015 18:36:10 [engine.c:2675] Completed hash walk (thread 0) 07/Dec/2015 18:36:10 [nprobe.c:4758] Waiting to export queued buckets... [queue len=4] 07/Dec/2015 18:36:10 [engine.c:2541] Emitting Flow: [->][udp] 172.31.0.0:50101 -> 172.31.200.50:2055 [2 pkt/304 bytes][ifIdx 65535->65535][0.0 sec][init Unknown][AS: 0 -> 0] 07/Dec/2015 18:36:10 [engine.c:2541] Emitting Flow: [->][udp] 172.31.0.8:50101 -> 172.31.200.50:2056 [1 pkt/152 bytes][ifIdx 65535->65535][0.0 sec][init Unknown][AS: 0 -> 0] 07/Dec/2015 18:36:10 [engine.c:2541] Emitting Flow: [->][udp] 172.31.0.8:50103 -> 172.31.200.50:2056 [2 pkt/164 bytes][ifIdx 65535->65535][0.0 sec][init Unknown][AS: 0 -> 0] 07/Dec/2015 18:36:10 [engine.c:2541] Emitting Flow: [->][udp] 172.31.0.0:50103 -> 172.31.200.50:2055 [4 pkt/328 bytes][ifIdx 65535->65535][0.0 sec][init Unknown][AS: 0 -> 0] 07/Dec/2015 18:36:11 [nprobe.c:4773] Pending buckets have been exported... 07/Dec/2015 18:36:13 [engine.c:3293] Export thread terminated [exportQueue=0] 07/Dec/2015 18:36:13 [nprobe.c:4839] Flushing queued flows... 07/Dec/2015 18:36:13 [nprobe.c:4842] Freeing memory... 07/Dec/2015 18:36:13 [plugin.c:277] Terminating plugins. 07/Dec/2015 18:36:13 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec] 07/Dec/2015 18:36:13 [nprobe.c:4934] Still allocated 0 hash buckets 07/Dec/2015 18:36:13 [nprobe.c:2457] Processed packets: 12 (max bucket search: 0) 07/Dec/2015 18:36:13 [nprobe.c:2440] Fragment queue length: 0 07/Dec/2015 18:36:13 [nprobe.c:2466] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent] 07/Dec/2015 18:36:13 [nprobe.c:2473] Flow collection: [collected pkts: 0][processed flows: 0] 07/Dec/2015 18:36:13 [nprobe.c:2476] Flow drop stats: [0 bytes/0 pkts][0 flows] 07/Dec/2015 18:36:13 [nprobe.c:2481] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent] 07/Dec/2015 18:36:13 [nprobe.c:4947] Cleaning globals 07/Dec/2015 18:36:13 [nprobe.c:4967] nProbe terminated.

The desire is to emit the flow data contained WITHIN these frames, not the flow data ABOUT these frames. Using -n instead of --zmq does not work either. The net observed behavior is that nProbe is not seeing the flow data contained in the frames from the router.

lucaderi commented 8 years ago

The apps work exactly as you described. Please do

  1. start nprobe nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234 <<== note the --zmq (double dash)
  2. start ntopng ntopng -i tcp://127.0.0.1:1234
  3. send nProbe some flows Example: nprobe -i ~/pcap/http/http.pcap

You will see them appear in ntopng.

smerkal commented 8 years ago

That's just it, I don't. Nor do I see them if I export to an external collector. I know it should work and it did before.

I have tried it exactly as you describe above using the .pcap files I have attached, as well as using live flow data being sent from the router on port 2055.

For the latter, I see the flows from the router hitting the server by the thousand using tcpdump. Decoding them with Wireshark shows they are valid IPFIX flows on the correct port. But they never get picked up by nProbe that is listening on that port.

This all worked when we first tested it a couple of months ago, but after a fresh install in preparation for production use, it no longer does. I am willing to give you access to the server if it helps, we are at a loss as to what changed.

lucaderi commented 8 years ago

Please contact me via email next week and we'll see what we can do

Thanks

smerkal commented 8 years ago

OK. Thanks Luca. In the mean time I'll keep poking at it.

Erik

On Tue, Dec 8, 2015 at 8:36 AM, Luca Deri notifications@github.com wrote:

Please contact me via email next week and we'll see what we can do

Thanks

— Reply to this email directly or view it on GitHub https://github.com/ntop/nProbe/issues/10#issuecomment-162900619.

smerkal commented 8 years ago

Luca,

If you are still willing/able, we would appreciate any assistance you can provide. Let me know what you would need from us.

Erik

On Tue, Dec 8, 2015 at 8:36 AM, Luca Deri notifications@github.com wrote:

Please contact me via email next week and we'll see what we can do

Thanks

— Reply to this email directly or view it on GitHub https://github.com/ntop/nProbe/issues/10#issuecomment-162900619.

lucaderi commented 8 years ago

@ValentinaViscarelli Hi Valentina can help please?

ValentinaViscarelli commented 8 years ago

Smerkal, please do this: 1) check you don't have nprobe and ntopng instances run. 2) Open a shell and run this command: nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234 -b2 3) Open another shell and run this command: ntopng -i tcp://127.0.0.1:1234 4) Open another shell and run this command (use pcap that you attached): nprobe -i cflow.pcap -b2 5) wait a couple of minutes 6) send me the output of three commands.

Thanks

smerkal commented 8 years ago

Valentina,

This all works as expected. However, what I see in ntopng is flow data ABOUT the packets in the capture (two hosts exchanging flow data) not the flow data sent from the router that is contained WITHIN these packets (hundreds of hosts communicating with the internet). It's acting the same as if I were running -i while this .pcap file was being captured.

I am trying to get nprobe to receive the flow data that the router is sending on port 2055 and proxy it to ntopng or other collectors, but all I can get it to do is send flow data about the headers of the frames it is receiving from the router, not the data about the routers transient traffic that is contained within the frame.

I am not sure how else to describe it that makes more sense. I just know that it used to work and now does not.

Erik

root@uncsnbox:~# nprobe -i none -n none -32055 --zmq tcp://127.0.0.1:1234 -b2 15/Dec/2015 09:02:04 [nprobe.c:3130] Valid nProbe Pro license found 15/Dec/2015 09:02:04 [plugin.c:166] No plugins found in ./plugins 15/Dec/2015 09:02:04 [plugin.c:174] Loading 22 plugins [.so] from /usr/local/lib/nprobe/plugins 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin DHCP Protocol: missing license [/etc/nprobe.license.dhcp] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin Diameter Protocol: missing license [/etc/nprobe.license.diameter] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin DNS Protocol: missing license [/etc/nprobe.license.dns] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin Export Plugin: missing license [/etc/nprobe.license.export] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin FTP Protocol: missing license [/etc/nprobe.license.ftp] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin GTPv0 Signaling Protocol: missing license [/etc/nprobe.license.gtpv0] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin GTPv1 Signaling Protocol: missing license [/etc/nprobe.license.gtpv1] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin GTPv2 Signaling Protocol: missing license [/etc/nprobe.license.gtpv2] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin HTTP Protocol: missing license [/etc/nprobe.license.http] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin IMAP Protocol: missing license [/etc/nprobe.license.email] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin Netflow-Lite Plugin: missing license [/etc/nprobe.license.nflite] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin Oracle Protocol: missing license [/etc/nprobe.license.oracle] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin POP3 Protocol: missing license [/etc/nprobe.license.email] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin System process information: missing license [/etc/nprobe.license.process] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin Radius Protocol: missing license [/etc/nprobe.license.radius] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin RTP Plugin: missing license [/etc/nprobe.license.voippro] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin S1AP Protocol: missing license [/etc/nprobe.license.S1AP] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin SIP Plugin: missing license [/etc/nprobe.license.voippro] 15/Dec/2015 09:02:04 [plugin.c:742] Unable to enable plugin SMTP Protocol: missing license [/etc/nprobe.license.email] 15/Dec/2015 09:02:04 [nprobe.c:4488] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 15/Dec/2015 09:02:04 [nprobe.c:4491] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 15/Dec/2015 09:02:04 [nprobe.c:4552] Welcome to nProbe Pro v.7.2.151211 ($Revision: 4471 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration 15/Dec/2015 09:02:04 [nprobe.c:4562] Running on Ubuntu 14.04.2 LTS 15/Dec/2015 09:02:04 [nprobe.c:4573] [LICENSE] nProbe SystemId: FA623D157104A1D2 15/Dec/2015 09:02:04 [nprobe.c:4620] Tracing enabled 15/Dec/2015 09:02:04 [bgpPlugin.c:375] BGP plugin is disabled (--bgp-port has not been specified) 15/Dec/2015 09:02:04 [dbPlugin.c:49] Initializing DB plugin 15/Dec/2015 09:02:04 [mysqlPlugin.c:111] Initialized MySQL plugin 15/Dec/2015 09:02:04 [plugin.c:248] 3 plugin(s) loaded [3 delete][2 packet]. 15/Dec/2015 09:02:04 [nprobe.c:6526] Welcome to nprobe v.7.2.151211 for x86_64-unknown-linux-gnu 15/Dec/2015 09:02:04 [nprobe.c:5752] Compiling flow templates... 15/Dec/2015 09:02:04 [plugin.c:851] Scanning plugin BGP Update Listener [bgp] 15/Dec/2015 09:02:04 [plugin.c:851] Scanning plugin MySQL DB [db] 15/Dec/2015 09:02:04 [plugin.c:851] Scanning plugin MySQL Plugin [mysql] 15/Dec/2015 09:02:04 [plugin.c:1000] 0 plugin(s) enabled 15/Dec/2015 09:02:04 [nprobe.c:6203] Non IPv4/v6 traffic is discarded according to the template 15/Dec/2015 09:02:04 [util.c:287] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat 15/Dec/2015 09:02:04 [util.c:296] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat 15/Dec/2015 09:02:04 [nprobe.c:5121] Using packet capture length 128 15/Dec/2015 09:02:04 [nprobe.c:6698] IPv6 traffic will NOT be exported/accounted by this probe 15/Dec/2015 09:02:04 [nprobe.c:6699] due to configuration options (e.g. use NetFlow v9) 15/Dec/2015 09:02:04 [nprobe.c:6702] The flows hash has 131072 buckets 15/Dec/2015 09:02:04 [nprobe.c:6704] Flows older than 120 seconds will be exported 15/Dec/2015 09:02:04 [nprobe.c:6707] Flows inactive for at least 30 seconds will be exported 15/Dec/2015 09:02:04 [nprobe.c:6710] Expired flows will not be queued for more than 30 seconds 15/Dec/2015 09:02:04 [nprobe.c:6717] Exported flows with engineType 0 and engineId 108 15/Dec/2015 09:02:04 [nprobe.c:6739] TCP TOS will be ignored and set to 0. 15/Dec/2015 09:02:04 [nprobe.c:6757] After 1 flow packets are sent, we'll delay at least 1 ms 15/Dec/2015 09:02:04 [nprobe.c:6777] Flows will be emitted in NetFlow 5 format 15/Dec/2015 09:02:04 [nprobe.c:6807] Flow input interface index is set to 0 15/Dec/2015 09:02:04 [nprobe.c:6813] Flow output interface index is set to 0 15/Dec/2015 09:02:04 [nprobe.c:6827] Not capturing packet from interface (collector mode) 15/Dec/2015 09:02:04 [util.c:3840] Succesfully created ZMQ endpoint tcp:// 127.0.0.1:1234 15/Dec/2015 09:02:04 [plugin.c:813] Disabling plugin BGP Update Listener (no template is using it) 15/Dec/2015 09:02:04 [plugin.c:813] Disabling plugin MySQL DB (no template is using it) 15/Dec/2015 09:02:04 [plugin.c:813] Disabling plugin MySQL Plugin (no template is using it) 15/Dec/2015 09:02:04 [collect.c:86] Created UDP sockets 15/Dec/2015 09:02:04 [collect.c:90] Created a SCTP socket (102) 15/Dec/2015 09:02:04 [collect.c:145] Flow collector listening on port 2055 (IPv4/v6) 15/Dec/2015 09:02:04 [nprobe.c:6947] Starting 1 packet fetch thread(s) 15/Dec/2015 09:02:04 [nprobe.c:7035] nProbe started successfully 15/Dec/2015 09:02:04 [engine.c:3210] Starting bucket dequeue thread 15/Dec/2015 09:02:22 [engine.c:2361] New Flow: [udp] 172.31.0.0:50101 -> 172.31.200.50:2055 [00:00:00:00:00:00 -> 00:00:00:00:00:00][vlan 65535][tos 0][ifIdx: 0 -> 0][subflowId: 0/0x0000][idx=11126] 15/Dec/2015 09:02:22 [engine.c:2361] New Flow: [udp] 172.31.0.0:50103 -> 172.31.200.50:2055 [00:00:00:00:00:00 -> 00:00:00:00:00:00][vlan 65535][tos 0][ifIdx: 0 -> 0][subflowId: 0/0x0000][idx=11132] 15/Dec/2015 09:02:51 [util.c:3865] [ZMQ] {"8":"172.31.0.0","12":"172.31.200.50","15":"0.0.0.0","10":0,"14":0,"2":3257,"1":1399640,"22":1450191741,"21":1450191741,"7":50101,"11":2055,"6":0,"4":17,"5":0,"16":0,"17":0,"9":0,"13":0,"42":1} 15/Dec/2015 09:02:51 [engine.c:2541] Emitting Flow: [->][udp] 172.31.0.0:50101 -> 172.31.200.50:2055 [3257 pkt/1399640 bytes][ifIdx 0->0][0.0 sec][init Unknown][AS: 0 -> 0] 15/Dec/2015 09:02:51 [util.c:3865] [ZMQ] {"8":"172.31.0.0","12":"172.31.200.50","15":"0.0.0.0","10":0,"14":0,"2":99,"1":8112,"22":1450191741,"21":1450191741,"7":50103,"11":2055,"6":0,"4":17,"5":0,"16":0,"17":0,"9":0,"13":0,"42":2} 15/Dec/2015 09:02:51 [engine.c:2541] Emitting Flow: [->][udp] 172.31.0.0:50103 -> 172.31.200.50:2055 [99 pkt/8112 bytes][ifIdx 0->0][0.0 sec][init Unknown][AS: 0 -> 0]

root@uncsnbox:~# ntopng -i tcp://127.0.0.1:1234 15/Dec/2015 09:02:16 [Ntop.cpp:933] Setting local networks to 127.0.0.0/8 15/Dec/2015 09:02:16 [Redis.cpp:106] Successfully connected to redis 127.0.0.1:6379@0 15/Dec/2015 09:02:16 [NtopPro.cpp:119] [LICENSE] Reading license from /etc/ntopng.license 15/Dec/2015 09:02:16 [Ntop.cpp:1152] Registered interface tcp:// 127.0.0.1:1234 [id: 0] 15/Dec/2015 09:02:16 [Ntop.cpp:1165] Registered interface view tcp:// 127.0.0.1:1234 [id: 0] 15/Dec/2015 09:02:16 [Utils.cpp:304] User changed to nobody 15/Dec/2015 09:02:16 [main.cpp:240] PID stored in file /var/tmp/ntopng.pid 15/Dec/2015 09:02:16 [HTTPserver.cpp:465] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL. 15/Dec/2015 09:02:16 [HTTPserver.cpp:482] -->3000<-- 15/Dec/2015 09:02:16 [HTTPserver.cpp:510] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts] 15/Dec/2015 09:02:16 [HTTPserver.cpp:513] HTTP server listening on port 3000 15/Dec/2015 09:02:16 [main.cpp:290] Working directory: /var/tmp/ntopng 15/Dec/2015 09:02:16 [main.cpp:292] Scripts/HTML pages directory: /usr/share/ntopng 15/Dec/2015 09:02:16 [Ntop.cpp:260] Welcome to ntopng x86_64 v.2.2.151211 - (C) 1998-15 ntop.org 15/Dec/2015 09:02:16 [Ntop.cpp:265] Built on Ubuntu 14.04.2 LTS 15/Dec/2015 09:02:16 [PeriodicActivities.cpp:53] Started periodic activities loop... 15/Dec/2015 09:02:16 [RuntimePrefs.cpp:32] Dumping alerts into syslog 15/Dec/2015 09:02:16 [NtopPro.cpp:233] [LICENSE] ntopng systemId: FA623D157104A1D2 15/Dec/2015 09:02:16 [NtopPro.cpp:244] [LICENSE] ntopng license: 2163EA9A6D3FEBD13E0940ACB875D3DC1480454122251EC47F 15/Dec/2015 09:02:16 [NtopPro.cpp:265] [LICENSE] Maintenance is available until Tue Nov 29 15:15:22 2016 [350 days left] 15/Dec/2015 09:02:16 [NetworkInterface.cpp:1426] Started packet polling on interface tcp://127.0.0.1:1234 [id: 0]... 15/Dec/2015 09:02:17 [CollectorInterface.cpp:94] Collecting flows on tcp:// 127.0.0.1:1234

root@uncsnbox:~# nprobe -i cflow.pcap -b2 15/Dec/2015 09:02:21 [nprobe.c:3130] Valid nProbe Pro license found 15/Dec/2015 09:02:21 [plugin.c:166] No plugins found in ./plugins 15/Dec/2015 09:02:21 [plugin.c:174] Loading 22 plugins [.so] from /usr/local/lib/nprobe/plugins 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin DHCP Protocol: missing license [/etc/nprobe.license.dhcp] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin Diameter Protocol: missing license [/etc/nprobe.license.diameter] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin DNS Protocol: missing license [/etc/nprobe.license.dns] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin Export Plugin: missing license [/etc/nprobe.license.export] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin FTP Protocol: missing license [/etc/nprobe.license.ftp] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin GTPv0 Signaling Protocol: missing license [/etc/nprobe.license.gtpv0] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin GTPv1 Signaling Protocol: missing license [/etc/nprobe.license.gtpv1] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin GTPv2 Signaling Protocol: missing license [/etc/nprobe.license.gtpv2] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin HTTP Protocol: missing license [/etc/nprobe.license.http] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin IMAP Protocol: missing license [/etc/nprobe.license.email] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin Netflow-Lite Plugin: missing license [/etc/nprobe.license.nflite] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin Oracle Protocol: missing license [/etc/nprobe.license.oracle] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin POP3 Protocol: missing license [/etc/nprobe.license.email] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin System process information: missing license [/etc/nprobe.license.process] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin Radius Protocol: missing license [/etc/nprobe.license.radius] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin RTP Plugin: missing license [/etc/nprobe.license.voippro] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin S1AP Protocol: missing license [/etc/nprobe.license.S1AP] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin SIP Plugin: missing license [/etc/nprobe.license.voippro] 15/Dec/2015 09:02:21 [plugin.c:742] Unable to enable plugin SMTP Protocol: missing license [/etc/nprobe.license.email] 15/Dec/2015 09:02:21 [nprobe.c:4488] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 15/Dec/2015 09:02:21 [nprobe.c:4491] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 15/Dec/2015 09:02:21 [nprobe.c:4552] Welcome to nProbe Pro v.7.2.151211 ($Revision: 4471 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration 15/Dec/2015 09:02:21 [nprobe.c:4562] Running on Ubuntu 14.04.2 LTS 15/Dec/2015 09:02:21 [nprobe.c:4573] [LICENSE] nProbe SystemId: FA623D157104A1D2 15/Dec/2015 09:02:21 [nprobe.c:4620] Tracing enabled 15/Dec/2015 09:02:21 [nprobe.c:4658] WARNING: -n parameter is missing. 127.0.0.1:2055 will be used. 15/Dec/2015 09:02:21 [nprobe.c:2948] Exporting flows towards 127.0.0.1:2055 using UDP 15/Dec/2015 09:02:21 [bgpPlugin.c:375] BGP plugin is disabled (--bgp-port has not been specified) 15/Dec/2015 09:02:21 [dbPlugin.c:49] Initializing DB plugin 15/Dec/2015 09:02:21 [mysqlPlugin.c:111] Initialized MySQL plugin 15/Dec/2015 09:02:21 [plugin.c:248] 3 plugin(s) loaded [3 delete][2 packet]. 15/Dec/2015 09:02:21 [nprobe.c:6526] Welcome to nprobe v.7.2.151211 for x86_64-unknown-linux-gnu 15/Dec/2015 09:02:21 [nprobe.c:5752] Compiling flow templates... 15/Dec/2015 09:02:21 [plugin.c:851] Scanning plugin BGP Update Listener [bgp] 15/Dec/2015 09:02:21 [plugin.c:851] Scanning plugin MySQL DB [db] 15/Dec/2015 09:02:21 [plugin.c:851] Scanning plugin MySQL Plugin [mysql] 15/Dec/2015 09:02:21 [plugin.c:1000] 0 plugin(s) enabled 15/Dec/2015 09:02:21 [nprobe.c:6203] Non IPv4/v6 traffic is discarded according to the template 15/Dec/2015 09:02:21 [util.c:287] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat 15/Dec/2015 09:02:21 [util.c:296] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat 15/Dec/2015 09:02:21 [nprobe.c:5121] Using packet capture length 128 15/Dec/2015 09:02:21 [nprobe.c:6698] IPv6 traffic will NOT be exported/accounted by this probe 15/Dec/2015 09:02:21 [nprobe.c:6699] due to configuration options (e.g. use NetFlow v9) 15/Dec/2015 09:02:21 [nprobe.c:6702] The flows hash has 131072 buckets 15/Dec/2015 09:02:21 [nprobe.c:6704] Flows older than 120 seconds will be exported 15/Dec/2015 09:02:21 [nprobe.c:6707] Flows inactive for at least 30 seconds will be exported 15/Dec/2015 09:02:21 [nprobe.c:6710] Expired flows will not be queued for more than 30 seconds 15/Dec/2015 09:02:21 [nprobe.c:6717] Exported flows with engineType 0 and engineId 125 15/Dec/2015 09:02:21 [nprobe.c:6739] TCP TOS will be ignored and set to 0. 15/Dec/2015 09:02:21 [nprobe.c:6757] After 1 flow packets are sent, we'll delay at least 1 ms 15/Dec/2015 09:02:21 [nprobe.c:6777] Flows will be emitted in NetFlow 5 format 15/Dec/2015 09:02:21 [nprobe.c:6807] Flow input interface index is set to 0 15/Dec/2015 09:02:21 [nprobe.c:6813] Flow output interface index is set to 0 15/Dec/2015 09:02:21 [plugin.c:813] Disabling plugin BGP Update Listener (no template is using it) 15/Dec/2015 09:02:21 [plugin.c:813] Disabling plugin MySQL DB (no template is using it) 15/Dec/2015 09:02:21 [plugin.c:813] Disabling plugin MySQL Plugin (no template is using it) 15/Dec/2015 09:02:21 [nprobe.c:6947] Starting 1 packet fetch thread(s) 15/Dec/2015 09:02:21 [nprobe.c:5496] Fetch packets thread started [thread 0] 15/Dec/2015 09:02:21 [engine.c:3210] Starting bucket dequeue thread 15/Dec/2015 09:02:21 [engine.c:2361] New Flow: [udp] 172.31.0.0:50101 -> 172.31.200.50:2055 [00:00:00:00:00:00 -> 00:00:00:00:00:00][vlan 0][tos 0][ifIdx: 65535 -> 65535][subflowId: 0/0x0000][idx=76663] 15/Dec/2015 09:02:21 [engine.c:2361] New Flow: [udp] 172.31.0.0:50103 -> 172.31.200.50:2055 [00:00:00:00:00:00 -> 00:00:00:00:00:00][vlan 0][tos 0][ifIdx: 65535 -> 65535][subflowId: 0/0x0000][idx=76669] 15/Dec/2015 09:02:21 [nprobe.c:5592] fetchPcapPackets(): no more packets to read (capture file over?) 15/Dec/2015 09:02:21 [nprobe.c:5636] fetchPcapPackets(threadId=0) terminated 15/Dec/2015 09:02:21 [nprobe.c:7035] nProbe started successfully 15/Dec/2015 09:02:21 [nprobe.c:7044] No more packets to read. Sleeping... 15/Dec/2015 09:02:21 [nprobe.c:4716] nProbe is shutting down... 15/Dec/2015 09:02:21 [nprobe.c:4752] Exporting pending buckets... 15/Dec/2015 09:02:21 [engine.c:2673] About to flush hash (threadId 0) 15/Dec/2015 09:02:21 [engine.c:2675] Completed hash walk (thread 0) 15/Dec/2015 09:02:21 [nprobe.c:4758] Waiting to export queued buckets... [queue len=2] 15/Dec/2015 09:02:21 [engine.c:2541] Emitting Flow: [->][udp] 172.31.0.0:50101 -> 172.31.200.50:2055 [3257 pkt/1399640 bytes][ifIdx 0->0][0.0 sec][init Unknown][AS: 0 -> 0] 15/Dec/2015 09:02:21 [engine.c:2541] Emitting Flow: [->][udp] 172.31.0.0:50103 -> 172.31.200.50:2055 [99 pkt/8112 bytes][ifIdx 0->0][0.0 sec][init Unknown][AS: 0 -> 0] 15/Dec/2015 09:02:22 [export.c:1266] Sending 2 flows (NetFlow v5 format) 15/Dec/2015 09:02:22 [nprobe.c:4773] Pending buckets have been exported... 15/Dec/2015 09:02:24 [engine.c:3293] Export thread terminated [exportQueue=0] 15/Dec/2015 09:02:24 [nprobe.c:4839] Flushing queued flows... 15/Dec/2015 09:02:24 [nprobe.c:4842] Freeing memory... 15/Dec/2015 09:02:24 [plugin.c:277] Terminating plugins. 15/Dec/2015 09:02:24 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec] 15/Dec/2015 09:02:24 [nprobe.c:4934] Still allocated 0 hash buckets 15/Dec/2015 09:02:24 [nprobe.c:2457] Processed packets: 3356 (max bucket search: 0) 15/Dec/2015 09:02:24 [nprobe.c:2440] Fragment queue length: 0 15/Dec/2015 09:02:24 [nprobe.c:2466] Flow export stats: [1407752 bytes/3356 pkts][2 flows/1 pkts sent] 15/Dec/2015 09:02:24 [nprobe.c:2476] Flow drop stats: [0 bytes/0 pkts][0 flows] 15/Dec/2015 09:02:24 [nprobe.c:2481] Total flow stats: [1407752 bytes/3356 pkts][2 flows/1 pkts sent] 15/Dec/2015 09:02:24 [nprobe.c:4947] Cleaning globals 15/Dec/2015 09:02:24 [nprobe.c:4967] nProbe terminated.

On Tue, Dec 15, 2015 at 1:51 AM, ValentinaViscarelli < notifications@github.com> wrote:

Smerkal, please do this: 1) check you don't have nprobe and ntopng instances run. 2) Open a shell and run this command: nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234 -b2 3) Open another shell and run this command: ntopng -i tcp://127.0.0.1:1234 4) Open another shell and run this command (use pcap that you attached): nprobe -i cflow.pcap -b2 5) wait a couple of minutes 6) send me the output of three commands.

Thanks

— Reply to this email directly or view it on GitHub https://github.com/ntop/nProbe/issues/10#issuecomment-164676656.

ValentinaViscarelli commented 8 years ago

Smerkal,

this was just a try. I wanted to see if ntopng received something. it's normal you see only "two hosts exchanging flow data". I try to explain.... The nprobe instance at point 4 simulate your router; so if you use in input a pcap file with a IPFIX traffic it's wrong. Your router receive in input normal traffic and export IPFIX traffic; so you have to use in input a pcap with normal traffic. Try this and if it works, repeat the procedure without point 4 but with your router that exports IPFIX flows. If it doesn't work, please send me commands outputs and after, if possible, we can think a remote connection.

Thanks

smerkal commented 8 years ago

nProbe works just fine if using -i or feeding it a .pcap file of normal transient traffic captured from the router. It just does not work when acting as a proxy for the flow data received from the router. It acts like it doesn't see the data coming in, even though I can verify that the data is being received. I have tried sending it IPFIX from Juniper and Netflow v5 from Cisco at point 4 with the same results.

Erik

root@uncsnbox:/home/nbox# nprobe -i none -n none -3 2055 --zmq tcp:// 127.0.0.1:1234 -b 2 16/Dec/2015 13:35:17 [nprobe.c:3130] Valid nProbe Pro license found 16/Dec/2015 13:35:17 [plugin.c:166] No plugins found in ./plugins 16/Dec/2015 13:35:17 [plugin.c:174] Loading 22 plugins [.so] from /usr/local/lib/nprobe/plugins 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin DHCP Protocol: missing license [/etc/nprobe.license.dhcp] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin Diameter Protocol: missing license [/etc/nprobe.license.diameter] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin DNS Protocol: missing license [/etc/nprobe.license.dns] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin Export Plugin: missing license [/etc/nprobe.license.export] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin FTP Protocol: missing license [/etc/nprobe.license.ftp] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin GTPv0 Signaling Protocol: missing license [/etc/nprobe.license.gtpv0] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin GTPv1 Signaling Protocol: missing license [/etc/nprobe.license.gtpv1] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin GTPv2 Signaling Protocol: missing license [/etc/nprobe.license.gtpv2] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin HTTP Protocol: missing license [/etc/nprobe.license.http] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin IMAP Protocol: missing license [/etc/nprobe.license.email] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin Netflow-Lite Plugin: missing license [/etc/nprobe.license.nflite] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin Oracle Protocol: missing license [/etc/nprobe.license.oracle] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin POP3 Protocol: missing license [/etc/nprobe.license.email] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin System process information: missing license [/etc/nprobe.license.process] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin Radius Protocol: missing license [/etc/nprobe.license.radius] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin RTP Plugin: missing license [/etc/nprobe.license.voippro] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin S1AP Protocol: missing license [/etc/nprobe.license.S1AP] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin SIP Plugin: missing license [/etc/nprobe.license.voippro] 16/Dec/2015 13:35:17 [plugin.c:742] Unable to enable plugin SMTP Protocol: missing license [/etc/nprobe.license.email] 16/Dec/2015 13:35:17 [nprobe.c:4488] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 16/Dec/2015 13:35:17 [nprobe.c:4491] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 16/Dec/2015 13:35:17 [nprobe.c:4552] Welcome to nProbe Pro v.7.2.151211 ($Revision: 4471 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration 16/Dec/2015 13:35:17 [nprobe.c:4562] Running on Ubuntu 14.04.2 LTS 16/Dec/2015 13:35:17 [nprobe.c:4573] [LICENSE] nProbe SystemId: FA623D157104A1D2 16/Dec/2015 13:35:17 [nprobe.c:4620] Tracing enabled 16/Dec/2015 13:35:17 [bgpPlugin.c:375] BGP plugin is disabled (--bgp-port has not been specified) 16/Dec/2015 13:35:17 [dbPlugin.c:49] Initializing DB plugin 16/Dec/2015 13:35:17 [mysqlPlugin.c:111] Initialized MySQL plugin 16/Dec/2015 13:35:17 [plugin.c:248] 3 plugin(s) loaded [3 delete][2 packet]. 16/Dec/2015 13:35:17 [nprobe.c:6526] Welcome to nprobe v.7.2.151211 for x86_64-unknown-linux-gnu 16/Dec/2015 13:35:17 [nprobe.c:5752] Compiling flow templates... 16/Dec/2015 13:35:17 [plugin.c:851] Scanning plugin BGP Update Listener [bgp] 16/Dec/2015 13:35:17 [plugin.c:851] Scanning plugin MySQL DB [db] 16/Dec/2015 13:35:17 [plugin.c:851] Scanning plugin MySQL Plugin [mysql] 16/Dec/2015 13:35:17 [plugin.c:1000] 0 plugin(s) enabled 16/Dec/2015 13:35:17 [nprobe.c:6203] Non IPv4/v6 traffic is discarded according to the template 16/Dec/2015 13:35:17 [util.c:287] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat 16/Dec/2015 13:35:17 [util.c:296] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat 16/Dec/2015 13:35:17 [nprobe.c:5121] Using packet capture length 128 16/Dec/2015 13:35:17 [nprobe.c:6698] IPv6 traffic will NOT be exported/accounted by this probe 16/Dec/2015 13:35:17 [nprobe.c:6699] due to configuration options (e.g. use NetFlow v9) 16/Dec/2015 13:35:17 [nprobe.c:6702] The flows hash has 131072 buckets 16/Dec/2015 13:35:17 [nprobe.c:6704] Flows older than 120 seconds will be exported 16/Dec/2015 13:35:17 [nprobe.c:6707] Flows inactive for at least 30 seconds will be exported 16/Dec/2015 13:35:17 [nprobe.c:6710] Expired flows will not be queued for more than 30 seconds 16/Dec/2015 13:35:17 [nprobe.c:6717] Exported flows with engineType 0 and engineId 245 16/Dec/2015 13:35:17 [nprobe.c:6739] TCP TOS will be ignored and set to 0. 16/Dec/2015 13:35:17 [nprobe.c:6757] After 1 flow packets are sent, we'll delay at least 1 ms 16/Dec/2015 13:35:17 [nprobe.c:6777] Flows will be emitted in NetFlow 5 format 16/Dec/2015 13:35:17 [nprobe.c:6807] Flow input interface index is set to 0 16/Dec/2015 13:35:17 [nprobe.c:6813] Flow output interface index is set to 0 16/Dec/2015 13:35:17 [nprobe.c:6827] Not capturing packet from interface (collector mode) 16/Dec/2015 13:35:17 [util.c:3840] Succesfully created ZMQ endpoint tcp:// 127.0.0.1:1234 16/Dec/2015 13:35:17 [plugin.c:813] Disabling plugin BGP Update Listener (no template is using it) 16/Dec/2015 13:35:17 [plugin.c:813] Disabling plugin MySQL DB (no template is using it) 16/Dec/2015 13:35:17 [plugin.c:813] Disabling plugin MySQL Plugin (no template is using it) 16/Dec/2015 13:35:17 [collect.c:86] Created UDP sockets 16/Dec/2015 13:35:17 [collect.c:90] Created a SCTP socket (102) 16/Dec/2015 13:35:17 [collect.c:145] Flow collector listening on port 2055 (IPv4/v6) 16/Dec/2015 13:35:17 [nprobe.c:6947] Starting 1 packet fetch thread(s) 16/Dec/2015 13:35:17 [engine.c:3210] Starting bucket dequeue thread 16/Dec/2015 13:35:17 [nprobe.c:7035] nProbe started successfully ^C16/Dec/2015 13:37:16 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec] 16/Dec/2015 13:37:16 [nprobe.c:386] Received shutdown request... [signal: 2] 16/Dec/2015 13:37:16 [nprobe.c:4716] nProbe is shutting down... 16/Dec/2015 13:37:16 [nprobe.c:4752] Exporting pending buckets... 16/Dec/2015 13:37:16 [nprobe.c:4773] Pending buckets have been exported... 16/Dec/2015 13:37:18 [engine.c:3293] Export thread terminated [exportQueue=0] 16/Dec/2015 13:37:18 [nprobe.c:4839] Flushing queued flows... 16/Dec/2015 13:37:18 [nprobe.c:4842] Freeing memory... 16/Dec/2015 13:37:18 [plugin.c:277] Terminating plugins. 16/Dec/2015 13:37:18 [cache.c:1200] Redis Cache [0 total/0.0 get/sec][0 total/0.0 set/sec] 16/Dec/2015 13:37:18 [nprobe.c:4934] Still allocated 0 hash buckets 16/Dec/2015 13:37:18 [nprobe.c:2457] Processed packets: 0 (max bucket search: 0) 16/Dec/2015 13:37:18 [nprobe.c:2440] Fragment queue length: 0 16/Dec/2015 13:37:18 [nprobe.c:2466] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent] 16/Dec/2015 13:37:18 [nprobe.c:2473] Flow collection: [collected pkts: 0][processed flows: 0] 16/Dec/2015 13:37:18 [nprobe.c:2476] Flow drop stats: [0 bytes/0 pkts][0 flows] 16/Dec/2015 13:37:18 [nprobe.c:2481] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent] 16/Dec/2015 13:37:18 [nprobe.c:4947] Cleaning globals 16/Dec/2015 13:37:18 [nprobe.c:4967] nProbe terminated.

root@uncsnbox:/home/nbox# ntopng -i tcp://127.0.0.1:1234 16/Dec/2015 13:35:27 [Ntop.cpp:933] Setting local networks to 127.0.0.0/8 16/Dec/2015 13:35:27 [Redis.cpp:106] Successfully connected to redis 127.0.0.1:6379@0 16/Dec/2015 13:35:27 [NtopPro.cpp:119] [LICENSE] Reading license from /etc/ntopng.license 16/Dec/2015 13:35:27 [Ntop.cpp:1152] Registered interface tcp:// 127.0.0.1:1234 [id: 0] 16/Dec/2015 13:35:27 [Ntop.cpp:1165] Registered interface view tcp:// 127.0.0.1:1234 [id: 0] 16/Dec/2015 13:35:27 [Utils.cpp:304] User changed to nobody 16/Dec/2015 13:35:27 [main.cpp:240] PID stored in file /var/tmp/ntopng.pid 16/Dec/2015 13:35:27 [HTTPserver.cpp:465] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL. 16/Dec/2015 13:35:27 [HTTPserver.cpp:482] -->3000<-- 16/Dec/2015 13:35:27 [HTTPserver.cpp:510] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts] 16/Dec/2015 13:35:27 [HTTPserver.cpp:513] HTTP server listening on port 3000 16/Dec/2015 13:35:27 [main.cpp:290] Working directory: /var/tmp/ntopng 16/Dec/2015 13:35:27 [main.cpp:292] Scripts/HTML pages directory: /usr/share/ntopng 16/Dec/2015 13:35:27 [Ntop.cpp:260] Welcome to ntopng x86_64 v.2.2.151211 - (C) 1998-15 ntop.org 16/Dec/2015 13:35:27 [Ntop.cpp:265] Built on Ubuntu 14.04.2 LTS 16/Dec/2015 13:35:27 [PeriodicActivities.cpp:53] Started periodic activities loop... 16/Dec/2015 13:35:27 [RuntimePrefs.cpp:32] Dumping alerts into syslog 16/Dec/2015 13:35:27 [NtopPro.cpp:233] [LICENSE] ntopng systemId: FA623D157104A1D2 16/Dec/2015 13:35:27 [NtopPro.cpp:244] [LICENSE] ntopng license: 2163EA9A6D3FEBD13E0940ACB875D3DC1480454122251EC47F 16/Dec/2015 13:35:27 [NtopPro.cpp:265] [LICENSE] Maintenance is available until Tue Nov 29 15:15:22 2016 [349 days left] 16/Dec/2015 13:35:27 [NetworkInterface.cpp:1426] Started packet polling on interface tcp://127.0.0.1:1234 [id: 0]... 16/Dec/2015 13:35:28 [CollectorInterface.cpp:94] Collecting flows on tcp:// 127.0.0.1:1234 ^C16/Dec/2015 13:37:10 [main.cpp:37] Shutting down... 16/Dec/2015 13:37:12 [ProtoStats.cpp:35] [IPv4] 0 B/0.00 Packets 16/Dec/2015 13:37:12 [ProtoStats.cpp:35] [IPv6] 0 B/0.00 Packets 16/Dec/2015 13:37:12 [ProtoStats.cpp:35] [ARP] 0 B/0.00 Packets 16/Dec/2015 13:37:12 [ProtoStats.cpp:35] [MPLS] 0 B/0.00 Packets 16/Dec/2015 13:37:12 [ProtoStats.cpp:35] [Other] 0 B/0.00 Packets 16/Dec/2015 13:37:13 [Ntop.cpp:1191] Interface tcp://127.0.0.1:1234 [running: 0] 16/Dec/2015 13:37:13 [main.cpp:48] Deleted PID /var/tmp/ntopng.pid [rc: 0] 16/Dec/2015 13:37:13 [HTTPserver.cpp:525] HTTP server terminated 16/Dec/2015 13:37:13 [AddressResolution.cpp:54] Address resolution stats [0 resolved][0 failures]

root@uncsnbox:/home/nbox# tcpdump -n -l -i p1p1 port 2055 13:36:52.134060 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.135056 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.136078 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.137066 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.138039 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.139069 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.140067 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.141067 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.142079 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.143057 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.144053 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.145054 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.146059 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.147057 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.148059 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.149054 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.150058 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.151056 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.152057 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.153056 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.154054 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.155056 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.156057 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.157057 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.158057 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.159057 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.160056 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.161056 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420 13:36:52.162056 IP 172.31.0.0.50101 > 172.31.200.50.2055: UDP, length 420

On Wed, Dec 16, 2015 at 3:32 AM, ValentinaViscarelli < notifications@github.com> wrote:

Smerkal,

this was just a try. I wanted to see if ntopng received something. it's normal you see only "two hosts exchanging flow data". I try to explain.... The nprobe instance at point 4 simulate your router; so if you use in input a pcap file with a IPFIX traffic it's wrong. Your router receive in input normal traffic and export IPFIX traffic; so you have to use in input a pcap with normal traffic. Try this and if it works, repeat the procedure without point 4 but with your router that exports IPFIX flows. If it doesn't work, please send me commands outputs and after, if possible, we can think a remote connection.

Thanks

— Reply to this email directly or view it on GitHub https://github.com/ntop/nProbe/issues/10#issuecomment-165046160.

ValentinaViscarelli commented 8 years ago

Erik, I think you have a problem with firewall.... I believe that firewall allows traffic on port 2055 from localhost and denies traffic from remote. You see the traffic with tcpdump because it acts on level 2 but then the traffic is blocked by "iptables". Please try these two scenarios with netcat:

Scenario 1: 1) run this command: netcat -ul 2055 | hexdump -C 2) run nprobe: nprobe -i file.pcap

Scenario 2: 1) run this command: netcat -ul 2055 | hexdump -C 2) send flow data with router

If you have problem with firewall in "Scenario 2" you should receive no bytes on netcat.

smerkal commented 8 years ago

I see nothing in scenario 2, UFW (iptables) is disabled.

On Wed, Dec 16, 2015 at 4:32 PM, ValentinaViscarelli < notifications@github.com> wrote:

Erik, I think you have a problem with firewall.... I believe that firewall allows traffic on port 2055 from localhost and denies traffic from remote. You see the traffic with tcpdump because it acts on level 2 but then the traffic is blocked by "iptables". Please try these two scenarios with netcat:

Scenario 1: 1) run this command: netcat -ul 2055 | hexdump -C 2) run nprobe: nprobe -i file.pcap

Scenario 2: 1) run this command: netcat -ul 2055 | hexdump -C 2) send flow data with router

If you have problem with firewall in "Scenario 2" you should receive no bytes on netcat.

— Reply to this email directly or view it on GitHub https://github.com/ntop/nProbe/issues/10#issuecomment-165268866.

smerkal commented 8 years ago

Resolved. Apparently Ubuntu is doing RPF checks. Flow data was coming in on one interface but the route back to the source of the flow data was out another interface (default gateway) and traffic was being rejected. Adding a route to the flow source through the interface that it is being received on (or disabling rp_filter) resolved the issue.

Thank you for the assistance and I apologize for wasting your time.

On Wed, Dec 16, 2015 at 5:25 PM, Erik Schmersal erik@schmersal.us wrote:

I see nothing in scenario 2, UFW (iptables) is disabled.

On Wed, Dec 16, 2015 at 4:32 PM, ValentinaViscarelli < notifications@github.com> wrote:

Erik, I think you have a problem with firewall.... I believe that firewall allows traffic on port 2055 from localhost and denies traffic from remote. You see the traffic with tcpdump because it acts on level 2 but then the traffic is blocked by "iptables". Please try these two scenarios with netcat:

Scenario 1: 1) run this command: netcat -ul 2055 | hexdump -C 2) run nprobe: nprobe -i file.pcap

Scenario 2: 1) run this command: netcat -ul 2055 | hexdump -C 2) send flow data with router

If you have problem with firewall in "Scenario 2" you should receive no bytes on netcat.

— Reply to this email directly or view it on GitHub https://github.com/ntop/nProbe/issues/10#issuecomment-165268866.

ValentinaViscarelli commented 8 years ago

Hi Erik, no problem. Don't hesitate to contact us if you have other problems.

Cheers, Valentina