search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.26k stars 656 forks source link

ntopng dumps to elasticsearch with incorrect timestamp #199

Closed kthkaya closed 9 years ago

kthkaya commented 9 years ago

Only 3 out of 400.000 dumped flows have the correct timestamp.

root@nTop:/etc/ntopng# cat ntopng.conf -w=3000 -W=0 -m="192.168.0.0/24" -F=es;flows;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;password -d=/storage/ntopng -G=/var/tmp/ntopng.pid -i=tcp://127.0.0.1:5556

Some dumped flows

October 5th 2015, 02:45:17.000 @timestamp:October 5th 2015, 02:45:17.000 type:flows IPV4_SRC_ADDR:140.205.159.56 L4_SRC_PORT:0 IPV4_DST_ADDR:0.0.0.0 L4_DST_PORT:0 PROTOCOL:6 TCP_FLAGS:0 IN_PKTS:5 IN_BYTES:405 OUT_PKTS:0 OUT_BYTES:0 FIRST_SWITCHED:1,440,789,636 LAST_SWITCHED:1,440,789,636 json.5:0 json.9:0 json.10:0 json.13:0 json.14:0 json.15:0.0.0.0 json.16:0 json.17:0 json.42:531762 CLIENT_NW_LATENCY_MS:0 SERVER_NW_LATENCY_MS:0 _id:AVA1dbrJKInKjCg_uSAI _type:flows _index:ntopng-2015.10.05

October 5th 2015, 02:15:05.000 @timestamp:October 5th 2015, 02:15:05.000 type:flows IPV4_SRC_ADDR:121.214.55.2 L4_SRC_PORT:0 IPV4_DST_ADDR:0.0.0.0 L4_DST_PORT:0 PROTOCOL:17 L7_PROTO:0 L7_PROTO_NAME:Unknown IN_PKTS:36 IN_BYTES:1,779 OUT_PKTS:0 OUT_BYTES:0 FIRST_SWITCHED:1,440,789,635 LAST_SWITCHED:1,440,789,636 json.5:0 json.9:0 json.10:0 json.13:0 json.14:0 json.15:0.0.0.0 json.16:1221 json.17:0 json.42:258897 SRC_IP_COUNTRY:AU SRC_IP_LOCATION:144.967, -37.817 _id:AVA1Wg8kKInKjCg_tfu9 _type:flows _index:ntopng-2015.10.05

October 5th 2015, 02:05:09.000 @timestamp:October 5th 2015, 02:05:09.000 type:flows IPV4_SRC_ADDR:62.210.205.156 L4_SRC_PORT:0 IPV4_DST_ADDR:0.0.0.0 L4_DST_PORT:0 PROTOCOL:6 L7_PROTO:0 L7_PROTO_NAME:Unknown TCP_FLAGS:0 IN_PKTS:104 IN_BYTES:146,553 OUT_PKTS:0 OUT_BYTES:0 FIRST_SWITCHED:1,440,789,635 LAST_SWITCHED:1,440,789,635 json.5:0 json.9:0 json.10:0 json.13:0 json.14:0 json.15:0.0.0.0 json.16:12876 json.17:0 json.42:170919 CLIENT_NW_LATENCY_MS:0 SERVER_NW_LATENCY_MS:0 SRC_IP_COUNTRY:FR SRC_IP_LOCATION:2.333, 48.867 _id:AVA1UPf4KInKjCg_tPkX _type:flows _index:ntopng-2015.10.05

August 28th 2015, 21:20:36.000 @timestamp:August 28th 2015, 21:20:36.000 type:flows IPV4_SRC_ADDR:46.188.121.201 L4_SRC_PORT:0 IPV4_DST_ADDR:0.0.0.0 L4_DST_PORT:0 PROTOCOL:17 IN_PKTS:1 IN_BYTES:99 OUT_PKTS:0 OUT_BYTES:0 FIRST_SWITCHED:1,440,789,636 LAST_SWITCHED:1,440,789,636 json.5:0 json.9:0 json.10:0 json.13:0 json.14:0 json.15:0.0.0.0 json.16:0 json.17:0 json.42:965 _id:AVA1QBnEKInKjCg_sw0B _type:flows _index:ntopng-2015.10.04

August 28th 2015, 21:20:36.000 @timestamp:August 28th 2015, 21:20:36.000 type:flows IPV4_SRC_ADDR:203.215.120.85 L4_SRC_PORT:0 IPV4_DST_ADDR:0.0.0.0 L4_DST_PORT:0 PROTOCOL:17 IN_PKTS:1 IN_BYTES:48 OUT_PKTS:0 OUT_BYTES:0 FIRST_SWITCHED:1,440,789,636 LAST_SWITCHED:1,440,789,636 json.5:0 json.9:0 json.10:0 json.13:0 json.14:0 json.15:0.0.0.0 json.16:6648 json.17:0 json.42:724 SRC_IP_COUNTRY:PH SRC_IP_LOCATION:121.05, 14.633 _id:AVA1QBnsKInKjCg_sw0T _type:flows _index:ntopng-2015.10.04

lmangani commented 9 years ago

Are all flows @timestamp fields wrong after the issue starts, or does it drifts back to current ts?

kthkaya commented 9 years ago

Yes, all flows are timestamped with 28th of Aug from the very beginning of dumps. So actually issue starts as dumps start. However every 15-20 minutes, one or two flows are timestamped with Oct 5 which is correct whereas the remaining thousands of flows are 28th

lmangani commented 9 years ago

Will try to replicate could you provide your full and exact system details.

lucaderi commented 9 years ago

If you collect flows, the timestamp is coming from your netflow flows. Can you please check if such timestamp is correct?

kthkaya commented 9 years ago

nTop nBox 2.3

Linux kernel 3.16.0-30-generic x86_64 CEST timezone ntp.ubuntu.com hwclock shows correct time

1x Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz

CPU 0 0
1x Intel Corporation 82540EM Gigabit Ethernet Controller

Ntopng Version: 2.0.151004

Nprobe Version: 7.2.151004

root@nTop:/etc/ntopng# cat ntopng.conf -w=3000 -W=0 -m="192.168.0.0/24" -F=es;flows;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;password -d=/storage/ntopng -G=/var/tmp/ntopng.pid -i=tcp://127.0.0.1:5556

root@nTop:/etc/nprobe# cat nprobe-none.conf -n=none -i=none -t=60 -d=60 -a=0 -e=1 -B=10 -w=128000 -z=0 -S=1:1 -E=0:0 -g=/var/run/nprobe-none.pid -3=2056 --zmq=tcp://*:5556 --vlanid-as-iface-idx=none -V=9 --dump-stats=/var/log/nprobe/none-0_flows_stats.txt

elasticsearch.yml network.host: localhost

root@nTop:/opt/kibana/config# cat kibana.yml

Kibana is served by a back end server. This controls which port to use.

port: 5601

The host to bind the server to.

host: "localhost"

The Elasticsearch instance to use for all your queries.

elasticsearch_url: "http://localhost:9200"

root@nTop:/etc/nginx/sites-available# cat default server { listen 5000;

server_name ntop;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/htpasswd.users;

location / {
    proxy_pass http://localhost:5601;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
}

}

Some additional info Kibana Index Pattern = ntopng* Mapping=@timestamp

lucaderi commented 9 years ago

@kthkaya Sorry the info you posted does not help. Again: can you check the timestamp inside your ingress flows if it;s correct? I think the problem is there.

kthkaya commented 9 years ago

@lmangani

I ran ntop with -b=2 in CLI and following are some flows

05/Oct/2015 04:33:35 [util.c:3923] [ZMQ] {"8":"183.15.136.248","12":"0.0.0.0","15":"0.0.0.0","10":0,"14":0,"2":1,"1":177,"22":4293539231,"21":1444012413,"7":0,"11":0,"6":0,"4":17,"5":0,"16":0,"17":0,"9":0,"13":0,"42":6120} 05/Oct/2015 04:33:35 [engine.c:2486] Emitting Flow: [->][udp] 183.15.136.248:0 -> 0.0.0.0:0 [1 pkt/177 bytes][ifIdx 0->0][0.0 sec][init Unknown][AS: 0 -> 0] 05/Oct/2015 04:33:35 [util.c:3923] [ZMQ]

{"8":"77.205.58.187","12":"0.0.0.0","15":"0.0.0.0","10":0,"14":0,"2":2,"1":293,"22":4293538199,"21":1444012413,"7":0,"11":0,"6":0,"4":6,"5":0,"16":15557,"17":0,"9":0,"13":0,"42":6121} 05/Oct/2015 04:33:35 [engine.c:2486] Emitting Flow: [->][tcp] 77.205.58.187:0 -> 0.0.0.0:0 [2 pkt/293 bytes][ifIdx 0->0][0.0 sec][init Unknown][AS: 15557 -> 0] 05/Oct/2015 04:33:35 [util.c:3923] [ZMQ]

{"8":"216.69.185.47","12":"0.0.0.0","15":"0.0.0.0","10":0,"14":0,"2":1,"1":154,"22":4293539231,"21":1444012413,"7":0,"11":0,"6":0,"4":17,"5":0,"16":26496,"17":0,"9":0,"13":0,"42":6122} 05/Oct/2015 04:33:35 [engine.c:2486] Emitting Flow: [->][udp] 216.69.185.47:0 -> 0.0.0.0:0 [1 pkt/154 bytes][ifIdx 0->0][0.0 sec][init Unknown][AS: 26496 -> 0]

So they seem correct.

Or do you mean the timestamp is actually passed as a Flow Information Element within the flow?

lmangani commented 9 years ago

@kthkaya as @lucaderi requested could you please check what timestamp is being sent by your devices to ntopng?

kthkaya commented 9 years ago

That is what I don't understand. The exporter has no means (as far as I know) for passing a timestamp.

lucaderi commented 9 years ago

Please send me a pcap (full packet size) with incoming flows collected by nProbe and I will tell you the answer to my questions, otherwise we're stuck here

kthkaya commented 9 years ago

sent via email

kthkaya commented 9 years ago

I switched to using logstash with netflow codec instead of nprobe. I am not having timestamp issue anymore