search

ntop / nProbe

Open source components and extensions for nProbe
http://ntop.org
GNU General Public License v2.0
1.66k stars 44 forks source link

When including most _OUT fields in V9 flow generation mode the bytes/packet values are incorrect #29

Closed DanEllis197 closed 8 years ago

DanEllis197 commented 8 years ago

If one uses the following -T string you will have very low values for in_pkts and in_bytes. Removing %OUT_PKTS, %OUT_BYTES, %RETRANSMITTED_OUT_PKTS and %OOORDER_OUT_PKTS fixes the issue and results in proper data.

-T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %IPV6_SRC_ADDR %IPV6_DST_ADDR %IN_SRC_MAC %OUT_DST_MAC %FRAGMENTS %CLIENT_NW_DELAY_USEC %CLIENT_NW_DELAY_MS %SERVER_NW_DELAY_USEC %SERVER_NW_DELAY_MS %APPL_LATENCY_MS %RETRANSMITTED_IN_PKTS %RETRANSMITTED_OUT_PKTS %OOORDER_IN_PKTS %OOORDER_OUT_PKTS”

lucaderi commented 8 years ago

I have tried to reproduce this issue using a pcap file. IPV4_SRC_ADDR|IPV4_DST_ADDR|INPUT_SNMP|OUTPUT_SNMP|IN_PKTS|IN_BYTES|OUT_PKTS|OUT_BYTES|FIRST_SWITCHED|LAST_SWITCHED|L4_SRC_PORT|L4_DST_PORT|TCP_FLAGS|PROTOCOL|SRC_TOS|IPV6_SRC_ADDR|IPV6_DST_ADDR|IN_SRC_MAC|OUT_DST_MAC|FRAGMENTS|APPL_LATENCY_MS|RETRANSMITTED_IN_PKTS|RETRANSMITTED_OUT_PKTS|OOORDER_IN_PKTS|OOORDER_OUT_PKTS 192.168.1.233|192.168.99.1|0|0|336|13881|525|758335|1455142159|1455142159|2645|143|31|6|0|::|::|00:90:F5:09:B0:27|00:04:75:B5:B4:97|0|0.000|0|0|0|0

and

IPV4_SRC_ADDR|IPV4_DST_ADDR|INPUT_SNMP|OUTPUT_SNMP|IN_PKTS|IN_BYTES|FIRST_SWITCHED|LAST_SWITCHED|L4_SRC_PORT|L4_DST_PORT|TCP_FLAGS|PROTOCOL|SRC_TOS|IPV6_SRC_ADDR|IPV6_DST_ADDR|IN_SRC_MAC|OUT_DST_MAC|FRAGMENTS|APPL_LATENCY_MS|RETRANSMITTED_IN_PKTS|RETRANSMITTED_OUT_PKTS|OOORDER_IN_PKTS|OOORDER_OUT_PKTS 192.168.1.233|192.168.99.1|0|0|336|13881|1455142202|1455142202|2645|143|31|6|0|::|::|00:90:F5:09:B0:27|00:04:75:B5:B4:97|0|0.000|0|0|0|0

so it looks to me the value for IN_PKTS/IN_BYTES do not change. How can I reproduce this bug (full command line)?

DanEllis197 commented 8 years ago

Commands below, and a graph adding the total traffic for each flows. In previous runs the "not good" was sending flows with a small amount of bytes/packets in the flows. Now I'm not seeing any "valid" flows with the template having data in in_bytes or out_bytes. I didn't pcap the flow data and examine it, can if you need.

Good: ./nprobes -n 208.76.14.242:20013 --lifetime-timeout 15 --idle-timeout 15 --queue-timeout 15 --sample-rate 1:16 --flow-version 9 --no-promisc -Q 11 -u 10 -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %IPV6_SRC_ADDR %IPV6_DST_ADDR %IN_SRC_MAC %OUT_DST_MAC %FRAGMENTS %CLIENT_NW_LATENCY_MS %SERVER_NW_LATENCY_MS %APPL_LATENCY_MS %RETRANSMITTED_IN_PKTS %OOORDER_IN_PKTS"

Not good: ./nprobes -n 208.76.14.242:20013 --lifetime-timeout 15 --idle-timeout 15 --queue-timeout 15 --sample-rate 1:16 --flow-version 9 --no-promisc -Q 11 -u 10 -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %IPV6_SRC_ADDR %IPV6_DST_ADDR %IN_SRC_MAC %OUT_DST_MAC %FRAGMENTS %CLIENT_NW_LATENCY_MS %SERVER_NW_LATENCY_MS %APPL_LATENCY_MS %RETRANSMITTED_IN_PKTS %RETRANSMITTED_OUT_PKTS %OOORDER_IN_PKTS %OOORDER_OUT_PKTS"

Output from starting good: root@novia2:~# ./nprobes -n 208.76.14.242:20013 --lifetime-timeout 15 --idle-timeout 15 --queue-timeout 15 --sample-rate 1:16 --flow-version 9 --no-promisc -Q 11 -u 10 -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %IPV6_SRC_ADDR %IPV6_DST_ADDR %IN_SRC_MAC %OUT_DST_MAC %FRAGMENTS %CLIENT_NW_LATENCY_MS %SERVER_NW_LATENCY_MS %APPL_LATENCY_MS %RETRANSMITTED_IN_PKTS %OOORDER_IN_PKTS" -i eth2 11/Feb/2016 01:35:02 [nprobe.c:3182] Valid nProbe Pro license found 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin SIP Plugin: missing license [/etc/nprobe.license.voippro] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin RTP Plugin: missing license [/etc/nprobe.license.voippro] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin HTTP Protocol: missing license [/etc/nprobe.license.http] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin SMTP Protocol: missing license [/etc/nprobe.license.email] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin Netflow-Lite Plugin: missing license [/etc/nprobe.license.nflite] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin DNS/LLMNR Protocol: missing license [/etc/nprobe.license.dns] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin Oracle Protocol: missing license [/etc/nprobe.license.oracle] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin GTPv0 Signaling Protocol: missing license [/etc/nprobe.license.gtpv0] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin GTPv1 Signaling Protocol: missing license [/etc/nprobe.license.gtpv1] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin GTPv2 Signaling Protocol: missing license [/etc/nprobe.license.gtpv2] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin Radius Protocol: missing license [/etc/nprobe.license.radius] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin DHCP Protocol: missing license [/etc/nprobe.license.dhcp] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin IMAP Protocol: missing license [/etc/nprobe.license.email] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin POP3 Protocol: missing license [/etc/nprobe.license.email] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin System process information: missing license [/etc/nprobe.license.process] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin Diameter Protocol: missing license [/etc/nprobe.license.diameter] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin S1AP Protocol: missing license [/etc/nprobe.license.S1AP] 11/Feb/2016 01:35:02 [plugin.c:744] Unable to enable plugin Export Plugin: missing license [/etc/nprobe.license.export] 11/Feb/2016 01:35:02 [nprobe.c:4679] Welcome to nProbe Pro v.7.3.151219 ($Revision: 4748 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration 11/Feb/2016 01:35:02 [nprobe.c:4689] Running on Debian GNU/Linux 8.2 (jessie) 11/Feb/2016 01:35:02 [nprobe.c:4700] [LICENSE] nProbe SystemId: 7EF340067104A1D2 11/Feb/2016 01:35:02 [nprobe.c:6707] Welcome to nProbe v.7.3.151219 for x86_64-unknown-linux-gnu 11/Feb/2016 01:35:02 [nprobe.c:5965] Using NetFlow Packet Payload Len: 1472 11/Feb/2016 01:35:02 [plugin.c:1007] 0 plugin(s) enabled 11/Feb/2016 01:35:02 [nprobe.c:6362] Each flow is 93 bytes long 11/Feb/2016 01:35:02 [nprobe.c:6363] The # packets per flow has been set to 14 11/Feb/2016 01:35:02 [nprobe.c:5251] Using packet capture length 128 11/Feb/2016 01:35:02 [nprobe.c:6926] Flows ASs will not be computed (missing GeoIP support) 11/Feb/2016 01:35:02 [nprobe.c:7011] Capturing packets from interface eth2 [snaplen: 128 bytes] 11/Feb/2016 01:35:02 [util.c:3091] nProbe changed user to 'nobody' 11/Feb/2016 01:35:02 [nprobe.c:7221] nProbe started successfully 11/Feb/2016 01:36:31 [nprobe.c:2512] Flow export stats: [102114217 bytes/38808 pkts][131 flows/12 pkts sent] 11/Feb/2016 01:36:31 [nprobe.c:2522] Flow drop stats: [159674 bytes/995 pkts][0 flows] 11/Feb/2016 01:36:31 [nprobe.c:2527] Total flow stats: [102273891 bytes/39803 pkts][131 flows/12 pkts sent]

Output from starting not good: root@novia2:~# ./nprobes -n 208.76.14.242:20013 --lifetime-timeout 15 --idle-timeout 15 --queue-timeout 15 --sample-rate 1:16 --flow-version 9 --no-promisc -Q 11 -u 10 -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %IPV6_SRC_ADDR %IPV6_DST_ADDR %IN_SRC_MAC %OUT_DST_MAC %FRAGMENTS %CLIENT_NW_LATENCY_MS %SERVER_NW_LATENCY_MS %APPL_LATENCY_MS %RETRANSMITTED_IN_PKTS %RETRANSMITTED_OUT_PKTS %OOORDER_IN_PKTS %OOORDER_OUT_PKTS" 11/Feb/2016 01:41:15 [nprobe.c:3182] Valid nProbe Pro license found 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin SIP Plugin: missing license [/etc/nprobe.license.voippro] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin RTP Plugin: missing license [/etc/nprobe.license.voippro] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin HTTP Protocol: missing license [/etc/nprobe.license.http] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin SMTP Protocol: missing license [/etc/nprobe.license.email] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin Netflow-Lite Plugin: missing license [/etc/nprobe.license.nflite] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin DNS/LLMNR Protocol: missing license [/etc/nprobe.license.dns] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin Oracle Protocol: missing license [/etc/nprobe.license.oracle] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin GTPv0 Signaling Protocol: missing license [/etc/nprobe.license.gtpv0] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin GTPv1 Signaling Protocol: missing license [/etc/nprobe.license.gtpv1] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin GTPv2 Signaling Protocol: missing license [/etc/nprobe.license.gtpv2] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin Radius Protocol: missing license [/etc/nprobe.license.radius] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin DHCP Protocol: missing license [/etc/nprobe.license.dhcp] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin IMAP Protocol: missing license [/etc/nprobe.license.email] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin POP3 Protocol: missing license [/etc/nprobe.license.email] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin System process information: missing license [/etc/nprobe.license.process] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin Diameter Protocol: missing license [/etc/nprobe.license.diameter] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin S1AP Protocol: missing license [/etc/nprobe.license.S1AP] 11/Feb/2016 01:41:15 [plugin.c:744] Unable to enable plugin Export Plugin: missing license [/etc/nprobe.license.export] 11/Feb/2016 01:41:15 [nprobe.c:4679] Welcome to nProbe Pro v.7.3.151219 ($Revision: 4748 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration 11/Feb/2016 01:41:15 [nprobe.c:4689] Running on Debian GNU/Linux 8.2 (jessie) 11/Feb/2016 01:41:15 [nprobe.c:4700] [LICENSE] nProbe SystemId: 7EF340067104A1D2 11/Feb/2016 01:41:15 [nprobe.c:6707] Welcome to nProbe v.7.3.151219 for x86_64-unknown-linux-gnu 11/Feb/2016 01:41:15 [nprobe.c:5965] Using NetFlow Packet Payload Len: 1472 11/Feb/2016 01:41:15 [plugin.c:1007] 0 plugin(s) enabled 11/Feb/2016 01:41:15 [nprobe.c:6362] Each flow is 109 bytes long 11/Feb/2016 01:41:15 [nprobe.c:6363] The # packets per flow has been set to 12 11/Feb/2016 01:41:15 [nprobe.c:5251] Using packet capture length 128 11/Feb/2016 01:41:16 [nprobe.c:6926] Flows ASs will not be computed (missing GeoIP support) 11/Feb/2016 01:41:16 [nprobe.c:7011] Capturing packets from interface eth2 [snaplen: 128 bytes] 11/Feb/2016 01:41:16 [util.c:3091] nProbe changed user to 'nobody' 11/Feb/2016 01:41:16 [nprobe.c:7221] nProbe started successfully 11/Feb/2016 01:42:32 [nprobe.c:2486] Fragment queue length: 0 11/Feb/2016 01:42:32 [nprobe.c:2512] Flow export stats: [4450677 bytes/51925 pkts][128 flows/15 pkts sent] 11/Feb/2016 01:42:32 [nprobe.c:2522] Flow drop stats: [292617 bytes/1923 pkts][0 flows] 11/Feb/2016 01:42:32 [nprobe.c:2527] Total flow stats: [4743294 bytes/53848 pkts][128 flows/15 pkts sent]

Dan Ellis, (m) 610-349-9017

Kentik.com http://kentik.com

On Wed, Feb 10, 2016 at 2:12 PM, Luca Deri notifications@github.com wrote:

I have tried to reproduce this issue using a pcap file.

IPV4_SRC_ADDR|IPV4_DST_ADDR|INPUT_SNMP|OUTPUT_SNMP|IN_PKTS|IN_BYTES|OUT_PKTS|OUT_BYTES|FIRST_SWITCHED|LAST_SWITCHED|L4_SRC_PORT|L4_DST_PORT|TCP_FLAGS|PROTOCOL|SRC_TOS|IPV6_SRC_ADDR|IPV6_DST_ADDR|IN_SRC_MAC|OUT_DST_MAC|FRAGMENTS|APPL_LATENCY_MS|RETRANSMITTED_IN_PKTS|RETRANSMITTED_OUT_PKTS|OOORDER_IN_PKTS|OOORDER_OUT_PKTS

192.168.1.233|192.168.99.1|0|0|336|13881|525|758335|1455142159|1455142159|2645|143|31|6|0|::|::|00:90:F5:09:B0:27|00:04:75:B5:B4:97|0|0.000|0|0|0|0

and

IPV4_SRC_ADDR|IPV4_DST_ADDR|INPUT_SNMP|OUTPUT_SNMP|IN_PKTS|IN_BYTES|FIRST_SWITCHED|LAST_SWITCHED|L4_SRC_PORT|L4_DST_PORT|TCP_FLAGS|PROTOCOL|SRC_TOS|IPV6_SRC_ADDR|IPV6_DST_ADDR|IN_SRC_MAC|OUT_DST_MAC|FRAGMENTS|APPL_LATENCY_MS|RETRANSMITTED_IN_PKTS|RETRANSMITTED_OUT_PKTS|OOORDER_IN_PKTS|OOORDER_OUT_PKTS

192.168.1.233|192.168.99.1|0|0|336|13881|1455142202|1455142202|2645|143|31|6|0|::|::|00:90:F5:09:B0:27|00:04:75:B5:B4:97|0|0.000|0|0|0|0

so it looks to me the value for IN_PKTS/IN_BYTES do not change. How can I reproduce this bug (full command line)?

— Reply to this email directly or view it on GitHub https://github.com/ntop/nProbe/issues/29#issuecomment-182602826.

ValentinaViscarelli commented 8 years ago

I see you use a nprobe version of December. I already fixed a similar bug in January. Please update to new version and try again.

DanEllis197 commented 8 years ago

Closing this as it's likely we should be using direction and in_bytes/packets vs in_bytes/packets and out_bytes/packets.