netflow timestamp changed to a wrong datetime

ulf-f commented 8 years ago

Hi, we collect netflow from an Cisco Adaptive Security Appliance Software Version 8.2(2). we have nprobe pro with elasticsearch export plugin.

configuration: --local-networks="10.43.0.0/16" -g=/var/run/nprobe.pid --flow-version=9 --dont-reforge-timestamps --event-log=/var/log/nprobe/nprobe.log --elastic="flows;nprobe-%Y.%m.%d;http://10.134.80.16:9200/_bulk;" --collector-port=40050 -i=none -n=none --online-license-check -b=1

If we start nprobe, the first 2 to up to 6 flows which are exported to elasticsearch have the correct timestamp and look fine after this the timestamp is wrong. For example: "@timestamp": "2016-07-14T13:38:16.0Z", changed to: "@timestamp":"2016-07-01T08:31:25Z",

We also discovered that sometimes in between there is one flow which will be exported with correct timestamp:

Long version example:

Correct timestamp: {"IPV4_SRC_ADDR":"10.152.50.27","IPV4_DST_ADDR":"10.132.50.33","IPV4_NEXT_HOP":"0.0.0.0","INPUT_SNMP":18,"OUTPUT_SNMP":14,"IN_PKTS":1,"IN_BYTES":315,"FIRST_SWITCHED":1467361884,"LAST_SWITCHED":10365934,"L4_SRC_PORT":53,"L4_DST_PORT":58989,"TCP_FLAGS":0,"PROTOCOL":17,"SRC_TOS":0,"SRC_AS":0,"DST_AS":0,"IPV4_SRC_MASK":0,"IPV4_DST_MASK":0,"@version":"1","@timestamp":"2016-07-14T15:10:48Z", "EXPORTER_IPV4_ADDRESS":"0.0.0.0"}

Wrong timestamp: {"IPV4_SRC_ADDR":"10.152.24.211","IPV4_DST_ADDR":"10.43.205.18","IPV4_NEXT_HOP":"0.0.0.0","INPUT_SNMP":18,"OUTPUT_SNMP":7,"IN_PKTS":1,"IN_BYTES":24,"FIRST_SWITCHED":1467361884,"LAST_SWITCHED":11155046,"L4_SRC_PORT":0,"L4_DST_PORT":23917,"TCP_FLAGS":0,"PROTOCOL":1,"SRC_TOS":0,"SRC_AS":0,"DST_AS":0,"IPV4_SRC_MASK":0,"IPV4_DST_MASK":0,"@version":"1","@timestamp":"2016-07-01T08:31:24Z", "EXPORTER_IPV4_ADDRESS":"0.0.0.0"}

Another Problem is that L4_SRC_PORT and L4_DST_PORT mixed.

We captured the firewall netflow exports and looked at them, timestamps are correct. I can send a capture to you if you would like to have a look.

thanks, ulf

ulf-f commented 8 years ago

Same Problem was seen with ntopng here https://github.com/ntop/ntopng/issues/199. We only run nprobe with export plugin and see the same problem there. I would like to avoid using logstash.

The version we run is maybe useful for you: nprobe -v

Build OS: CentOS Linux release 7.2.1511 (Core)

ulf-f commented 8 years ago

--dont-reforge-timestamps is not in the configuration anymore, it was only for testing but it doesn't changed anything regarding the problems described.

lucaderi commented 8 years ago

@ulf-f can you please send me via email a pcap containing flows and templates for reproducing the issue? All timestamps (XXXX_SWITCHED) don't seem to be good to me and I need to figure out what's inside the received flow that does not work.

lucaderi commented 8 years ago

@ulf-f I have just patched the nProbe code to handle flow records with invalid fields that apparently some Cisco devices are sending out. Please resync nProbe in about 30 mins (new packages are being built) and report if working now. If not, please attach a pcap file (or send it to me via email) containing template + flow so I can see what's going on in your case.

lucaderi commented 8 years ago

Closing for inactivity. Will reopen if necessary

ulf-f commented 8 years ago

Hi lucaderi,

sorry, missed your reply. Just send you pcap and screenshots. I will try the new nprobe code and let you know.

regards, ulf

ntop / nProbe

netflow timestamp changed to a wrong datetime #77