search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.26k stars 656 forks source link

ElasticSearch timestamp on flow-dump incorrect... #258

Closed dboehlke closed 8 years ago

dboehlke commented 9 years ago

The flow records being "flow-dump"ed to ElasticSearch have odd timestamps. Many of the host records shown in the "hosts/hosts" tabs also show "45 years, 319 days, 19 h, 50 min, 55 sec" under seen sense, other hosts have what I would consider to be a correct seen since, so this could be a problem with what is in my redist database. Both the "1970-01.01" and the "45 years, 319 days" would appear to be a "0" in the date/time field.

On a suggestion from @lmangani I check ntpd on both my servers, one running ntopng and the other running two instances of nProbe, and the times are in sync. Both probes are receiving sflow from switches in two of the data centers we operate out of.

curl 'localhost:9200/_cat/indices?v'
health status index             pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   ntopng-1970.01.01   5   1        152            0    611.6kb        611.6kb 
yellow open   ntopng-2015.11.04   5   1          4            0     64.8kb         64.8kb 
yellow open   ntopng-2015.11.05   5   1    2167560            0    755.5mb        755.5mb 
yellow open   ntopng-2015.11.01   5   1         56            0    301.9kb        301.9kb 
yellow open   .kibana             1   1          3            0     12.9kb         12.9kb 
yellow open   ntopng-2015.01.01   5   1        128            0    847.8kb        847.8kb

So when I run a search against today's records I am getting records in today's index with a timestamp from 1970. Although I have 2167560 records in "ntopng-2015.11.05" an ElasticSearch search query is only returning 5, which I assume are unique records, or probably an indication that I am still learning ElasticSearch. :-)

curl 'localhost:9200/ntopng-2015.11.05/_search?pretty'
{
  "took" : 31,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2323567,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "ntopng-2015.11.05",
      "_type" : "ntopng",
      "_id" : "AVDYnZtMaPfKFbT_UFcv",
      "_score" : 1.0,
      "_source":{ "@timestamp": "1970-01-01T00:00:00.0Z", "type": "ntopng", "IPV4_SRC_ADDR": "162.222.40.205", "L4_SRC_PORT": 443, "IPV4_DST_ADDR": "50.20.221.77", "L4_DST_PORT": 50236, "PROTOCOL": 6, "L7_PROTO": 91, "L7_PROTO_NAME": "SSL", "TCP_FLAGS": 16, "IN_PKTS": 40000, "IN_BYTES": 1600000, "OUT_PKTS": 0, "OUT_BYTES": 0, "FIRST_SWITCHED": 0, "LAST_SWITCHED": 0, "json": { "15": "0.0.0.0", "10": "580", "14": "652", "152": "1446743182000", "153": "1446743182000", "5": "0", "16": "62715", "17": "17184", "9": "0", "13": "0", "42": "13395682" }, "CLIENT_NW_LATENCY_MS": 0.000000, "SERVER_NW_LATENCY_MS": 0.000000, "SRC_IP_COUNTRY": "US", "SRC_IP_LOCATION": [ -93.216599, 44.975899 ], "DST_IP_COUNTRY": "US", "DST_IP_LOCATION": [ -118.309799, 34.061901 ], "PASS_VERDICT": true }
    }, {
      "_index" : "ntopng-2015.11.05",
      "_type" : "ntopng",
      "_id" : "AVDYnZtMaPfKFbT_UFcy",
      "_score" : 1.0,
      "_source":{ "@timestamp": "1970-01-01T00:00:00.0Z", "type": "ntopng", "IPV4_SRC_ADDR": "65.50.38.200", "L4_SRC_PORT": 49777, "IPV4_DST_ADDR": "162.222.41.22", "L4_DST_PORT": 443, "PROTOCOL": 6, "L7_PROTO": 91, "L7_PROTO_NAME": "SSL", "TCP_FLAGS": 16, "IN_PKTS": 40000, "IN_BYTES": 60000000, "OUT_PKTS": 0, "OUT_BYTES": 0, "FIRST_SWITCHED": 0, "LAST_SWITCHED": 0, "json": { "15": "0.0.0.0", "10": "589", "14": "526", "152": "1446743182000", "153": "1446743182000", "5": "0", "16": "31939", "17": "62715", "9": "0", "13": "0", "42": "13395693" }, "CLIENT_NW_LATENCY_MS": 0.000000, "SERVER_NW_LATENCY_MS": 0.000000, "SRC_IP_COUNTRY": "US", "SRC_IP_LOCATION": [ -84.377098, 33.771900 ], "DST_IP_COUNTRY": "US", "DST_IP_LOCATION": [ -93.216599, 44.975899 ], "PASS_VERDICT": true }
    }, {
      "_index" : "ntopng-2015.11.05",
      "_type" : "ntopng",
      "_id" : "AVDYnZtNaPfKFbT_UFc0",
      "_score" : 1.0,
      "_source":{ "@timestamp": "1970-01-01T00:00:00.0Z", "type": "ntopng", "IPV4_SRC_ADDR": "96.95.243.225", "L4_SRC_PORT": 54820, "IPV4_DST_ADDR": "162.222.40.140", "L4_DST_PORT": 443, "PROTOCOL": 6, "L7_PROTO": 91, "L7_PROTO_NAME": "SSL", "TCP_FLAGS": 16, "IN_PKTS": 20000, "IN_BYTES": 11520000, "OUT_PKTS": 0, "OUT_BYTES": 0, "FIRST_SWITCHED": 0, "LAST_SWITCHED": 0, "json": { "15": "0.0.0.0", "10": "597", "14": "524", "152": "1446743184000", "153": "1446743184000", "5": "0", "16": "7922", "17": "62715", "9": "0", "13": "0", "42": "13395906" }, "CLIENT_NW_LATENCY_MS": 0.000000, "SERVER_NW_LATENCY_MS": 0.000000, "SRC_IP_COUNTRY": "US", "SRC_IP_LOCATION": [ -97.000000, 38.000000 ], "DST_IP_COUNTRY": "US", "DST_IP_LOCATION": [ -93.216599, 44.975899 ], "PASS_VERDICT": true }
    }, {
      "_index" : "ntopng-2015.11.05",
      "_type" : "ntopng",
      "_id" : "AVDYnZthaPfKFbT_UFc1",
      "_score" : 1.0,
      "_source":{ "@timestamp": "1970-01-01T00:00:00.0Z", "type": "ntopng", "IPV4_SRC_ADDR": "76.21.178.43", "L4_SRC_PORT": 57573, "IPV4_DST_ADDR": "162.222.41.28", "L4_DST_PORT": 443, "PROTOCOL": 6, "L7_PROTO": 91, "L7_PROTO_NAME": "SSL", "TCP_FLAGS": 16, "IN_PKTS": 0, "IN_BYTES": 0, "OUT_PKTS": 40000, "OUT_BYTES": 1600000, "FIRST_SWITCHED": 0, "LAST_SWITCHED": 0, "json": { "15": "0.0.0.0", "10": "579", "14": "589", "152": "1446743182000", "153": "1446743182000", "5": "0", "16": "62715", "17": "7922", "9": "0", "13": "0", "42": "13395746" }, "CLIENT_NW_LATENCY_MS": 0.000000, "SERVER_NW_LATENCY_MS": 0.000000, "SRC_IP_COUNTRY": "US", "SRC_IP_LOCATION": [ -77.086304, 38.826500 ], "DST_IP_COUNTRY": "US", "DST_IP_LOCATION": [ -93.216599, 44.975899 ], "PASS_VERDICT": true }
    }, {
      "_index" : "ntopng-2015.11.05",
      "_type" : "ntopng",
      "_id" : "AVDYnZtsaPfKFbT_UFdB",
      "_score" : 1.0,
      "_source":{ "@timestamp": "1970-01-01T00:00:00.0Z", "type": "ntopng", "IPV4_SRC_ADDR": "68.44.162.231", "L4_SRC_PORT": 54897, "IPV4_DST_ADDR": "162.222.40.222", "L4_DST_PORT": 443, "PROTOCOL": 6, "L7_PROTO": 91, "L7_PROTO_NAME": "SSL", "TCP_FLAGS": 24, "IN_PKTS": 280000, "IN_BYTES": 154560000, "OUT_PKTS": 0, "OUT_BYTES": 0, "FIRST_SWITCHED": 0, "LAST_SWITCHED": 0, "json": { "15": "0.0.0.0", "10": "597", "14": "583", "152": "1446743094000", "153": "1446743213000", "5": "72", "16": "7922", "17": "62715", "9": "0", "13": "0", "42": "13395859" }, "CLIENT_NW_LATENCY_MS": 0.000000, "SERVER_NW_LATENCY_MS": 0.000000, "SRC_IP_COUNTRY": "US", "SRC_IP_LOCATION": [ -86.088097, 39.679401 ], "DST_IP_COUNTRY": "US", "DST_IP_LOCATION": [ -93.216599, 44.975899 ], "PASS_VERDICT": true }
    }, {
      "_index" : "ntopng-2015.11.05",
      "_type" : "ntopng",
      "_id" : "AVDYnZt3aPfKFbT_UFdL",
      "_score" : 1.0,
      "_source":{ "@timestamp": "1970-01-01T00:00:00.0Z", "type": "ntopng", "IPV4_SRC_ADDR": "162.222.40.179", "L4_SRC_PORT": 443, "IPV4_DST_ADDR": "94.137.106.26", "L4_DST_PORT": 50599, "PROTOCOL": 6, "L7_PROTO": 91, "L7_PROTO_NAME": "SSL", "TCP_FLAGS": 16, "IN_PKTS": 40000, "IN_BYTES": 1600000, "OUT_PKTS": 0, "OUT_BYTES": 0, "FIRST_SWITCHED": 0, "LAST_SWITCHED": 0, "json": { "15": "0.0.0.0", "10": "579", "14": "652", "152": "1446743185000", "153": "1446743185000", "5": "0", "16": "62715", "17": "33885", "9": "0", "13": "0", "42": "13396005" }, "CLIENT_NW_LATENCY_MS": 0.000000, "SERVER_NW_LATENCY_MS": 0.000000, "SRC_IP_COUNTRY": "US", "SRC_IP_LOCATION": [ -93.216599, 44.975899 ], "DST_IP_COUNTRY": "SE", "DST_IP_LOCATION": [ 18.068600, 59.329399 ], "PASS_VERDICT": true }
    }, {
      "_index" : "ntopng-2015.11.05",
      "_type" : "ntopng",
      "_id" : "AVDYnZuBaPfKFbT_UFdP",
      "_score" : 1.0,
      "_source":{ "@timestamp": "1970-01-01T00:00:00.0Z", "type": "ntopng", "IPV4_SRC_ADDR": "104.200.154.82", "L4_SRC_PORT": 53124, "IPV4_DST_ADDR": "162.222.41.18", "L4_DST_PORT": 443, "PROTOCOL": 6, "L7_PROTO": 91, "L7_PROTO_NAME": "SSL", "TCP_FLAGS": 24, "IN_PKTS": 40000, "IN_BYTES": 55680000, "OUT_PKTS": 0, "OUT_BYTES": 0, "FIRST_SWITCHED": 0, "LAST_SWITCHED": 0, "json": { "15": "0.0.0.0", "10": "652", "14": "580", "152": "1446743187000", "153": "1446743187000", "5": "0", "16": "46562", "17": "62715", "9": "0", "13": "0", "42": "13396295" }, "CLIENT_NW_LATENCY_MS": 0.000000, "SERVER_NW_LATENCY_MS": 0.000000, "SRC_IP_COUNTRY": "US", "SRC_IP_LOCATION": [ -122.312302, 47.542000 ], "DST_IP_COUNTRY": "US", "DST_IP_LOCATION": [ -93.216599, 44.975899 ], "PASS_VERDICT": true }
    }, {
      "_index" : "ntopng-2015.11.05",
      "_type" : "ntopng",
      "_id" : "AVDYnZuBaPfKFbT_UFdR",
      "_score" : 1.0,
      "_source":{ "@timestamp": "1970-01-01T00:00:00.0Z", "type": "ntopng", "IPV4_SRC_ADDR": "104.179.244.79", "L4_SRC_PORT": 61097, "IPV4_DST_ADDR": "162.222.41.114", "L4_DST_PORT": 443, "PROTOCOL": 6, "L7_PROTO": 91, "L7_PROTO_NAME": "SSL", "TCP_FLAGS": 16, "IN_PKTS": 20000, "IN_BYTES": 11040000, "OUT_PKTS": 0, "OUT_BYTES": 0, "FIRST_SWITCHED": 0, "LAST_SWITCHED": 0, "json": { "15": "0.0.0.0", "10": "597", "14": "579", "152": "1446743187000", "153": "1446743187000", "5": "0", "16": "7018", "17": "62715", "9": "0", "13": "0", "42": "13396267" }, "CLIENT_NW_LATENCY_MS": 0.000000, "SERVER_NW_LATENCY_MS": 0.000000, "SRC_IP_COUNTRY": "US", "SRC_IP_LOCATION": [ -96.903198, 33.073502 ], "DST_IP_COUNTRY": "US", "DST_IP_LOCATION": [ -93.216599, 44.975899 ], "PASS_VERDICT": true }
    }, {
      "_index" : "ntopng-2015.11.05",
      "_type" : "ntopng",
      "_id" : "AVDYnZuJaPfKFbT_UFdV",
      "_score" : 1.0,
      "_source":{ "@timestamp": "1970-01-01T00:00:00.0Z", "type": "ntopng", "IPV4_SRC_ADDR": "65.30.48.191", "L4_SRC_PORT": 58091, "IPV4_DST_ADDR": "162.222.40.148", "L4_DST_PORT": 443, "PROTOCOL": 6, "L7_PROTO": 91, "L7_PROTO_NAME": "SSL", "TCP_FLAGS": 16, "IN_PKTS": 40000, "IN_BYTES": 22080000, "OUT_PKTS": 0, "OUT_BYTES": 0, "FIRST_SWITCHED": 0, "LAST_SWITCHED": 0, "json": { "15": "0.0.0.0", "10": "652", "14": "581", "152": "1446743187000", "153": "1446743187000", "5": "0", "16": "10796", "17": "62715", "9": "0", "13": "0", "42": "13396244" }, "CLIENT_NW_LATENCY_MS": 0.000000, "SERVER_NW_LATENCY_MS": 0.000000, "SRC_IP_COUNTRY": "US", "SRC_IP_LOCATION": [ -88.157898, 42.970901 ], "DST_IP_COUNTRY": "US", "DST_IP_LOCATION": [ -93.216599, 44.975899 ], "PASS_VERDICT": true }
    }, {
      "_index" : "ntopng-2015.11.05",
      "_type" : "ntopng",
      "_id" : "AVDYnZuJaPfKFbT_UFdX",
      "_score" : 1.0,
      "_source":{ "@timestamp": "1970-01-01T00:00:00.0Z", "type": "ntopng", "IPV4_SRC_ADDR": "72.174.137.60", "L4_SRC_PORT": 49159, "IPV4_DST_ADDR": "162.222.41.38", "L4_DST_PORT": 443, "PROTOCOL": 6, "L7_PROTO": 91, "L7_PROTO_NAME": "SSL", "TCP_FLAGS": 16, "IN_PKTS": 80000, "IN_BYTES": 44160000, "OUT_PKTS": 0, "OUT_BYTES": 0, "FIRST_SWITCHED": 0, "LAST_SWITCHED": 0, "json": { "15": "0.0.0.0", "10": "597", "14": "583", "152": "1446743135000", "153": "1446743182000", "5": "40", "16": "33588", "17": "62715", "9": "0", "13": "0", "42": "13395676" }, "CLIENT_NW_LATENCY_MS": 0.000000, "SERVER_NW_LATENCY_MS": 0.000000, "SRC_IP_COUNTRY": "US", "SRC_IP_LOCATION": [ -113.849701, 46.940201 ], "DST_IP_COUNTRY": "US", "DST_IP_LOCATION": [ -93.216599, 44.975899 ], "PASS_VERDICT": true }
    } ]
  }
}

Here is a screen dump showing the "seen since" records on the "Hosts/Hosts" page.

screen shot 2015-11-05 at 2 25 52 pm

I am running ntopng Professional v.2.1.151105 and nProbe v.7.3.151105 (r4661) for x86_64-unknown-linux-gnu.

Let me know if I can provide any other information that would be helpful.

Thanks.

Dan Boehlke

lmangani commented 9 years ago

Dan,

Please summarize your ntopng/nprobe relay setup for full understanding of the context.

dboehlke commented 9 years ago

I have two servers running a derivative of Ubuntu 12.0.4 Linux. They are identical servers, both are Dell servers with Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz (6 cores) and 32GB of ram. One server is running ntopng and ElasticSearch2 and Kibanna4. The second server is running two instances of nProbe running on one server, they are listening for sflow exports from two of the data centers, one nProbe for each data center.

I am running ntopng Professional v.2.1.151105 and nProbe v.7.3.151105 (r4661) for x86_64-unknown-linux-gnu.

Here is the configuration for ntopng:

/etc/ntopng/ntopng.conf

$ sudo cat /etc/ntopng/ntopng.conf
-G=/var/tmp/ntopng.pid
-d=/var/tmp/ntopng
-p=/etc/ntopng/protos.txt 
-i=tcp://10.60.59.14:5556
-i=tcp://10.60.59.14:5557
--dump-flows="es;ntopng;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;"
--dns-mode=1
--sticky-hosts=none
--disable-login=1
--local-networks="2606:4A80::/32,192.168.0.0/24,172.16.0.0/16,10.0.0.0/8,38.81.66.0/23,209.208.232.0/23,209.208.241.0/24,209.208.250.0/24,50.93.246.0/23,50.93.255.0/24,162.222.47.0/24,216.17.8.0/24,38.92.136.0/24,162.222.40.0/21,162.222.40.0/23,162.222.46.0/24,103.8.239.0/24,149.5.7.0/24"
-w=3000

Here are the configurations for the two nProbe instances:

nprobe-sea-sflow.conf

$ sudo cat nprobe-sea-sflow.conf 
# Code42 "Local networks"
--local-networks="2606:4A80::/32,192.168.0.0/24,172.16.0.0/16,10.0.0.0/8,38.81.66.0/23,209.208.232.0/23,209.208.241.0/24,209.208.250.0/24,50.93.246.0/23,50.93.255.0/24,162.222.47.0/24,216.17.8.0/24,38.92.136.0/24,162.222.40.0/21,162.222.40.0/23,162.222.46.0/24,103.8.239.0/24,149.5.7.0/24"

# Use IPFIX when exporting flows to another application, add record formats for IPv6:
-V=10

# pid file location
-g=/var/tmp/nprobe-sea-sflow.pid

# collector input port
--collector-port=6344

# zmq host
# uncomment if you want anyone to connect:
#--zmq="tcp://*.5556"
--zmq="tcp://10.60.59.14:5557"

# disable packet capture from interface:
-i=none
-n=none

nprobe-usi-sflow.conf

$ sudo cat nprobe-usi-sflow.conf 
# Code42 "Local networks"
--local-networks="2606:4A80::/32,192.168.0.0/24,172.16.0.0/16,10.0.0.0/8,38.81.66.0/23,209.208.232.0/23,209.208.241.0/24,209.208.250.0/24,50.93.246.0/23,50.93.255.0/24,162.222.47.0/24,216.17.8.0/24,38.92.136.0/24,162.222.40.0/21,162.222.40.0/23,162.222.46.0/24,103.8.239.0/24,149.5.7.0/24"

# Use IPFIX when exporting flows to another application, add record formats for IPv6:
-V=10

# pid file location
-g=/var/tmp/nprobe-usi-sflow.pid

# collector input port
--collector-port=6343

# zmq host
# uncomment if you want anyone to connect:
#--zmq="tcp://*.5556"
--zmq="tcp://10.60.59.14:5556"

# disable packet capture from interface:
-i=none
-n=none

Let me know if there is anything else you would like to know about my configuration. Thanks.

lmangani commented 9 years ago

@dboehlke thanks for the detailed description - could you confirm timestamps are correct in the sflow exports? that seems the most likely cause for what you are experiencing

dboehlke commented 9 years ago

I had not considered that the switches are at fault.

Although one thing that causes me to doubt the switches as being at fault is that ntopng is sticking some flows in ElasticSearch under the ntopng-1970.01.01 index and others in todays index, but the timestamp field is from 1970.01.01 in both cases.

We are using Juniper EX-4550 switches as the source of the flow information. They are NTP synchronized and have the correct time and date. I cannot find a bug notification on sflow for these. I am working on capturing the sflow packets and decoding them. I should have more information in a hour or so. We have had a problem with SNMP returning bogus information, so a Junos bug/feature cannot be ruled out.

I should have more information in an hour or so.

Thanks.

dboehlke commented 9 years ago

I set up sflowtool so I could capture the sflow records that were intended for nProbe and got:

startDatagram =================================
datagramSourceIP 50.93.247.253
datagramSize 1308
unixSecondsUTC 1446842738
datagramVersion 5
agentSubId 17
agent 10.61.3.250
packetSequenceNo 45907
sysUpTime 3861370872
samplesInPacket 7
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 138468
sourceId 0:597
meanSkipCount 5000
samplePool 692335000
dropEvents 0
inputPort 520
outputPort 597
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 1522
strippedBytes 4
headerLen 128
headerBytes 54-E0-32-CA-D5-F0-3C-8A-B0-10-12-41-81-00-0F-A4-08-00-45-00-05-DC-75-6B-40-00-3F-06-CB-41-32-5D-F6-A0-A2-DE-29-93-00-16-90-8E-77-82-05-71-D7-A0-53-0C-50-10-03-11-C3-62-00-00-B1-90-FA-23-E2-D2-77-1F-A7-C8-E1-4E-BF-0E-82-46-9A-C3-77-23-86-96-7E-7C-3C-BE-F6-CF-38-DB-D6-09-4D-EA-17-1D-05-00-F9-06-FA-34-D7-B3-A1-82-78-99-B5-E2-21-35-BF-81-2D-8D-BC-0E-E9-90-DA-7A-2B-BA-A6-AB-1E-FE-F9-3C
dstMAC 54e032cad5f0
srcMAC 3c8ab0101241
decodedVLAN 4004
decodedPriority 0
IPSize 1500
ip.tot_len 1500
srcIP 50.93.246.160
dstIP 162.222.41.147
IPProtocol 6
IPTOS 0
IPTTL 63
TCPSrcPort 22
TCPDstPort 37006
TCPFlags 16
flowBlock_tag 0:1001
extendedType SWITCH
in_vlan 10
in_priority 0
out_vlan 4004
out_priority 0
endSample   ----------------------
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 123067
sourceId 0:552
meanSkipCount 5000
samplePool 615330000
dropEvents 0
inputPort 522
outputPort 552
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 64
strippedBytes 4
headerLen 60
headerBytes 00-13-5F-63-EC-00-3C-8A-B0-10-12-41-08-00-45-00-00-28-82-92-40-00-3F-06-0E-C1-D8-11-08-55-32-F5-97-21-01-BB-ED-EC-51-66-01-21-63-49-03-F4-50-10-08-02-53-E9-00-00-00-00-55-55-55-55
dstMAC 00135f63ec00
srcMAC 3c8ab0101241
IPSize 46
ip.tot_len 40
srcIP 216.17.8.85
dstIP 50.245.151.33
IPProtocol 6
IPTOS 0
IPTTL 63
TCPSrcPort 443
TCPDstPort 60908
TCPFlags 16
flowBlock_tag 0:1001
extendedType SWITCH
in_vlan 10
in_priority 0
out_vlan 4012
out_priority 0
endSample   ----------------------
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 138469
sourceId 0:597
meanSkipCount 5000
samplePool 692340000
dropEvents 0
inputPort 519
outputPort 597
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 1522
strippedBytes 4
headerLen 128
headerBytes 54-E0-32-CA-D7-F0-3C-8A-B0-10-12-41-81-00-0F-A2-08-00-45-00-05-DC-44-35-40-00-3F-06-FE-43-32-5D-F6-57-A2-DE-28-10-00-16-A6-BC-8F-6E-7B-10-94-74-7E-E0-50-10-01-75-AE-09-00-00-92-54-5D-4D-0F-6A-69-1C-26-03-37-85-CD-85-2B-52-C1-A9-DD-BF-C0-8E-C1-87-8B-BA-0F-FF-5D-01-B3-F1-05-55-35-B7-BF-AD-E6-25-45-0A-E5-DD-92-A4-0C-D9-E8-A5-5D-46-9F-F7-DE-F2-F7-94-90-2B-6B-1F-EA-5B-C6-A1-DB-2C-9F-3E
dstMAC 54e032cad7f0
srcMAC 3c8ab0101241
decodedVLAN 4002
decodedPriority 0
IPSize 1500
ip.tot_len 1500
srcIP 50.93.246.87
dstIP 162.222.40.16
IPProtocol 6
IPTOS 0
IPTTL 63
TCPSrcPort 22
TCPDstPort 42684
TCPFlags 16
flowBlock_tag 0:1001
extendedType SWITCH
in_vlan 10
in_priority 0
out_vlan 4002
out_priority 0
endSample   ----------------------
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 123068
sourceId 0:552
meanSkipCount 5000
samplePool 615335000
dropEvents 0
inputPort 521
outputPort 552
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 64
strippedBytes 4
headerLen 60
headerBytes 00-13-5F-63-EC-00-3C-8A-B0-10-12-41-08-00-45-00-00-28-63-8F-40-00-3F-06-78-A9-32-5D-FF-C3-CF-F2-5D-84-01-BB-87-9C-71-B1-BF-03-FC-C1-C2-E2-50-10-05-42-D1-49-00-00-00-00-55-55-55-55
dstMAC 00135f63ec00
srcMAC 3c8ab0101241
IPSize 46
ip.tot_len 40
srcIP 50.93.255.195
dstIP 207.242.93.132
IPProtocol 6
IPTOS 0
IPTTL 63
TCPSrcPort 443
TCPDstPort 34716
TCPFlags 16
flowBlock_tag 0:1001
extendedType SWITCH
in_vlan 30
in_priority 0
out_vlan 4012
out_priority 0
endSample   ----------------------
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 138470
sourceId 0:597
meanSkipCount 5000
samplePool 692345000
dropEvents 0
inputPort 521
outputPort 597
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 1522
strippedBytes 4
headerLen 128
headerBytes 54-E0-32-CA-D7-F0-3C-8A-B0-10-12-41-81-00-0F-A2-08-00-45-00-05-DC-BB-27-40-00-3F-06-CD-B4-D8-11-08-E7-A2-DE-29-69-A6-01-10-BB-90-07-F3-89-8F-93-1D-74-50-10-02-22-7A-5E-00-00-72-6A-4A-B5-35-F2-83-35-FC-F7-55-20-B7-AC-E6-11-23-1A-94-C5-26-F2-8C-6C-BF-EE-A4-24-02-55-19-51-78-71-B5-F3-D7-6E-E0-11-3A-18-7F-42-B8-53-0E-4D-4C-A1-36-E3-C8-A5-54-F2-4D-30-EE-45-B0-65-E2-B3-50-BB-FF-5F-8A-69
dstMAC 54e032cad7f0
srcMAC 3c8ab0101241
decodedVLAN 4002
decodedPriority 0
IPSize 1500
ip.tot_len 1500
srcIP 216.17.8.231
dstIP 162.222.41.105
IPProtocol 6
IPTOS 0
IPTTL 63
TCPSrcPort 42497
TCPDstPort 4283
TCPFlags 16
flowBlock_tag 0:1001
extendedType SWITCH
in_vlan 10
in_priority 0
out_vlan 4002
out_priority 0
endSample   ----------------------
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 1240
sourceId 0:597
counterBlock_tag 0:1
ifIndex 597
networkType 6
ifSpeed 10000000000
ifDirection 1
ifStatus 3
ifInOctets 1099206153280061
ifInUcastPkts 1807261508
ifInMulticastPkts 11877178
ifInBroadcastPkts 7364
ifInDiscards 0
ifInErrors 1373
ifInUnknownProtos 0
ifOutOctets 4506856743517002
ifOutUcastPkts 1882705725
ifOutMulticastPkts 10080449
ifOutBroadcastPkts 7415
ifOutDiscards 0
ifOutErrors 0
ifPromiscuousMode 0
counterBlock_tag 0:2
dot3StatsAlignmentErrors 1373
dot3StatsFCSErrors 1373
dot3StatsSingleCollisionFrames 0
dot3StatsMultipleCollisionFrames 0
dot3StatsSQETestErrors 0
dot3StatsDeferredTransmissions 0
dot3StatsLateCollisions 0
dot3StatsExcessiveCollisions 0
dot3StatsInternalMacTransmitErrors 0
dot3StatsCarrierSenseErrors 0
dot3StatsFrameTooLongs 0
dot3StatsInternalMacReceiveErrors 0
dot3StatsSymbolErrors 0
endSample   ----------------------
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 123069
sourceId 0:552
meanSkipCount 5000
samplePool 615340000
dropEvents 0
inputPort 517
outputPort 552
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 74
strippedBytes 4
headerLen 70
headerBytes 00-13-5F-63-EC-00-3C-8A-B0-10-12-41-08-00-45-00-00-38-C0-80-40-00-3F-06-88-1A-D8-11-08-03-63-02-B0-0E-01-BB-C0-25-78-47-DE-71-27-F2-50-4F-50-18-00-68-DA-25-00-00-FE-6F-4E-26-13-35-0A-C7-38-9D-44-54-39-71-30-39
dstMAC 00135f63ec00
srcMAC 3c8ab0101241
IPSize 56
ip.tot_len 56
srcIP 216.17.8.3
dstIP 99.2.176.14
IPProtocol 6
IPTOS 0
IPTTL 63
TCPSrcPort 443
TCPDstPort 49189
TCPFlags 24
flowBlock_tag 0:1001
extendedType SWITCH
in_vlan 10
in_priority 0
out_vlan 4012
out_priority 0
endSample   ----------------------
endDatagram   =================================

Just to be sure that the switch isn't flaking out from time to time, I ran a test filtering for just the "unixSecondsUTC" field:

$ sflowtool | grep unixSecondsUTC | head -100
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843402
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843403
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404
unixSecondsUTC 1446843404

This shows the "unixSecondsUTC" counting up nicely. I ran this for quite a while and only saw is counting up, no zeros.

My other question, now that I have seen the contents of the sflow record, is what is happening to the MAC addresses, since nProbe is getting them from sflow, but I only see 00:00:00:00:00:00 for all hosts in ntopng, This is probably something I should ask in another issue.

Thanks!

dboehlke commented 9 years ago

I pointed sysdig at the ZMQ connection between nProbe and ntopng. It appears the only timestamp fields exchanged are 153 and 153 which are flowstart and flowstop in milliseconds. There are no MAC addresses at this point. I watched this for quite a while and didn't see any zeros, however on most records, the two fields contain the same number, which depending on interpretation could imply that the flow ran for 0 seconds.

$ sudo sysdig -A -pc -c echo_fds fd.port=5556 | head -100
------ Read 535B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.246.48","12":"95.154.31.55","15":"0.0.0.0","10":519,"14":551,"2":195000,"1":292500000,"152":1446846073000,"153":1446846188000,"7":443,"11":55322,"6":16,"4":6,"5":0,"16":62715,"17":39642,"9":0,"13":0,"42":331132}(flowb{"8":"50.93.246.246","12":"104.129.194.76","15":"0.0.0.0","10":519,"14":551,"2":100000,"1":5200000,"152":1446846073000,"153":1446846188000,"7":443,"11":10924,"6":16,"4":6,"5":0,"16":62715,"17":22616,"9":0,"13":0,"42":331133}
------ Read 265B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.255.186","12":"71.183.205.130","15":"0.0.0.0","10":522,"14":551,"2":35000,"1":1400000,"152":1446846073000,"153":1446846183000,"7":443,"11":53629,"6":16,"4":6,"5":0,"16":62715,"17":701,"9":0,"13":0,"42":331134}
------ Read 266B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.247.102","12":"50.201.189.130","15":"0.0.0.0","10":522,"14":551,"2":80000,"1":4160000,"152":1446846073000,"153":1446846187000,"7":443,"11":50396,"6":16,"4":6,"5":0,"16":62715,"17":7922,"9":0,"13":0,"42":331135}
------ Read 261B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.11","12":"58.161.10.127","15":"0.0.0.0","10":517,"14":551,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":62961,"6":16,"4":6,"5":0,"16":62715,"17":1221,"9":0,"13":0,"42":331136}
------ Read 269B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.246.251","12":"162.222.41.247","15":"0.0.0.0","10":520,"14":577,"2":15000,"1":22500000,"152":1446846145000,"153":1446846163000,"7":4506,"11":34395,"6":16,"4":6,"5":0,"16":62715,"17":62715,"9":0,"13":0,"42":331137}
------ Read 260B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.7","12":"38.81.66.45","15":"0.0.0.0","10":517,"14":551,"2":10000,"1":3945000,"152":1446846142000,"153":1446846163000,"7":43067,"11":4283,"6":24,"4":6,"5":0,"16":62715,"17":174,"9":0,"13":0,"42":331138}
------ Read 261B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.4","12":"198.177.6.251","15":"0.0.0.0","10":517,"14":552,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":65246,"6":16,"4":6,"5":0,"16":62715,"17":63061,"9":0,"13":0,"42":331139}
------ Read 265B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.247.103","12":"147.26.109.123","15":"0.0.0.0","10":522,"14":597,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":59576,"6":16,"4":6,"5":0,"16":62715,"17":18777,"9":0,"13":0,"42":331140}
------ Read 263B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.5","12":"162.222.41.192","15":"0.0.0.0","10":517,"14":597,"2":5000,"1":280000,"152":1446846163000,"153":1446846163000,"7":39682,"11":4283,"6":24,"4":6,"5":0,"16":62715,"17":62715,"9":0,"13":0,"42":331141}
------ Read 266B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.246.118","12":"51.174.180.224","15":"0.0.0.0","10":520,"14":552,"2":10000,"1":400000,"152":1446846149000,"153":1446846163000,"7":443,"11":51276,"6":16,"4":6,"5":0,"16":62715,"17":29695,"9":0,"13":0,"42":331142}
------ Read 263B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.246.190","12":"38.81.66.250","15":"0.0.0.0","10":520,"14":552,"2":5000,"1":7500000,"152":1446846163000,"153":1446846163000,"7":44403,"11":4283,"6":16,"4":6,"5":0,"16":62715,"17":174,"9":0,"13":0,"42":331143}
------ Read 259B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.7","12":"78.43.41.89","15":"0.0.0.0","10":517,"14":552,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":36935,"6":16,"4":6,"5":0,"16":62715,"17":29562,"9":0,"13":0,"42":331144}
------ Read 263B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.7","12":"162.222.41.133","15":"0.0.0.0","10":517,"14":597,"2":5000,"1":280000,"152":1446846163000,"153":1446846163000,"7":50666,"11":4283,"6":24,"4":6,"5":0,"16":62715,"17":62715,"9":0,"13":0,"42":331145}
------ Read 260B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.255.208","12":"171.64.85.48","15":"0.0.0.0","10":521,"14":577,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":58953,"6":16,"4":6,"5":0,"16":62715,"17":32,"9":0,"13":0,"42":331146}
------ Read 265B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.255.189","12":"66.129.241.14","15":"0.0.0.0","10":522,"14":551,"2":10000,"1":400000,"152":1446846136000,"153":1446846163000,"7":443,"11":37431,"6":16,"4":6,"5":0,"16":62715,"17":14203,"9":0,"13":0,"42":331147}
------ Read 257B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.3","12":"71.63.37.4","15":"0.0.0.0","10":517,"14":551,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":54590,"6":16,"4":6,"5":0,"16":62715,"17":7922,"9":0,"13":0,"42":331148}
------ Read 264B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.48","12":"162.222.40.241","15":"0.0.0.0","10":518,"14":597,"2":5000,"1":280000,"152":1446846163000,"153":1446846163000,"7":45079,"11":4283,"6":24,"4":6,"5":0,"16":62715,"17":62715,"9":0,"13":0,"42":331149}
------ Read 266B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.255.248","12":"173.219.61.146","15":"0.0.0.0","10":520,"14":552,"2":20000,"1":800000,"152":1446846131000,"153":1446846163000,"7":443,"11":52538,"6":16,"4":6,"5":0,"16":62715,"17":19108,"9":0,"13":0,"42":331150}
------ Read 263B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.11","12":"162.222.42.30","15":"0.0.0.0","10":517,"14":552,"2":5000,"1":280000,"152":1446846163000,"153":1446846163000,"7":60104,"11":4283,"6":24,"4":6,"5":0,"16":62715,"17":62715,"9":0,"13":0,"42":331151}
------ Read 258B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.8","12":"38.81.66.54","15":"0.0.0.0","10":517,"14":552,"2":5000,"1":280000,"152":1446846163000,"153":1446846163000,"7":4283,"11":52404,"6":24,"4":6,"5":0,"16":62715,"17":174,"9":0,"13":0,"42":331152}
------ Read 263B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.7","12":"162.222.41.233","15":"0.0.0.0","10":517,"14":577,"2":5000,"1":240000,"152":1446846163000,"153":1446846163000,"7":35556,"11":4283,"6":24,"4":6,"5":0,"16":62715,"17":62715,"9":0,"13":0,"42":331153}
------ Read 265B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.8","12":"162.222.40.151","15":"0.0.0.0","10":517,"14":597,"2":20000,"1":4505000,"152":1446846097000,"153":1446846163000,"7":37332,"11":4283,"6":24,"4":6,"5":0,"16":62715,"17":62715,"9":0,"13":0,"42":331154}
------ Read 265B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.48","12":"162.222.40.108","15":"0.0.0.0","10":518,"14":597,"2":5000,"1":3705000,"152":1446846163000,"153":1446846163000,"7":38993,"11":4283,"6":24,"4":6,"5":0,"16":62715,"17":62715,"9":0,"13":0,"42":331155}
------ Read 259B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.8","12":"68.50.65.211","15":"0.0.0.0","10":517,"14":552,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":29682,"6":16,"4":6,"5":0,"16":62715,"17":7922,"9":0,"13":0,"42":331156}
------ Read 263B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.244","12":"81.141.117.182","15":"0.0.0.0","10":521,"14":552,"2":5000,"1":340000,"152":1446846163000,"153":1446846163000,"7":443,"11":57431,"6":16,"4":6,"5":0,"16":62715,"17":2856,"9":0,"13":0,"42":331157}
------ Read 261B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.11","12":"24.56.178.115","15":"0.0.0.0","10":517,"14":577,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":25453,"6":16,"4":6,"5":0,"16":62715,"17":4181,"9":0,"13":0,"42":331158}
------ Read 261B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.51","12":"108.27.236.221","15":"0.0.0.0","10":518,"14":551,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":64222,"6":16,"4":6,"5":0,"16":62715,"17":701,"9":0,"13":0,"42":331159}
------ Read 263B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.150","12":"109.150.190.25","15":"0.0.0.0","10":521,"14":551,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":59205,"6":16,"4":6,"5":0,"16":62715,"17":2856,"9":0,"13":0,"42":331160}
------ Read 265B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.246.219","12":"204.14.239.105","15":"0.0.0.0","10":519,"14":551,"2":5000,"1":260000,"152":1446846163000,"153":1446846163000,"7":443,"11":42083,"6":16,"4":6,"5":0,"16":62715,"17":14340,"9":0,"13":0,"42":331161}
------ Read 262B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"216.17.8.105","12":"130.58.83.102","15":"0.0.0.0","10":522,"14":577,"2":5000,"1":260000,"152":1446846163000,"153":1446846163000,"7":443,"11":52810,"6":16,"4":6,"5":0,"16":62715,"17":3782,"9":0,"13":0,"42":331162}
------ Read 526B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.255.188","12":"173.239.65.9","15":"0.0.0.0","10":522,"14":551,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":60543,"6":16,"4":6,"5":0,"16":62715,"17":32354,"9":0,"13":0,"42":331163}(flowb{"8":"216.17.8.8","12":"172.98.67.77","15":"0.0.0.0","10":517,"14":551,"2":40000,"1":55680000,"152":1446846101000,"153":1446846163000,"7":443,"11":51979,"6":16,"4":6,"5":0,"16":62715,"17":46562,"9":0,"13":0,"42":331164}
------ Read 263B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.246.24","12":"72.34.128.250","15":"0.0.0.0","10":519,"14":551,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":51911,"6":16,"4":6,"5":0,"16":62715,"17":10725,"9":0,"13":0,"42":331165}
------ Read 265B from  [host] [host]  10.60.59.15:39808->10.60.59.14:freeciv (ntopng)

(flowb{"8":"50.93.246.176","12":"64.183.234.242","15":"0.0.0.0","10":520,"14":552,"2":5000,"1":200000,"152":1446846163000,"153":1446846163000,"7":443,"11":41264,"6":16,"4":6,"5":0,"16":62715,"17":11427,"9":0,"13":0,"42":331166}

Could this be an issue with how I have nProbe configured? Here is one of my nProbe configs:

$ sudo cat nprobe-sea-sflow.conf 
# Code42 "Local networks"
--local-networks="2606:4A80::/32,192.168.0.0/24,172.16.0.0/16,10.0.0.0/8,38.81.66.0/23,209.208.232.0/23,209.208.241.0/24,209.208.250.0/24,50.93.246.0/23,50.93.255.0/24,162.222.47.0/24,216.17.8.0/24,38.92.136.0/24,162.222.40.0/21,162.222.40.0/23,162.222.46.0/24,103.8.239.0/24,149.5.7.0/24"

# Use IPFIX when exporting flows to another application, add record formats for IPv6:
-V=10

# pid file location
-g=/var/tmp/nprobe-sea-sflow.pid

# collector input port
--collector-port=6344

# zmq host
# uncomment if you want anyone to connect:
#--zmq="tcp://*.5556"
--zmq="tcp://10.60.59.14:5557"

# disable packet capture from interface:
-i=none
-n=none

I do get this warning on startup:

06/Nov/2015 14:59:34 [nprobe.c:5932] WARNING: You selected v9/IPFIX without specifying a template (-T).
06/Nov/2015 14:59:34 [nprobe.c:5933] WARNING: The default template will be used

Let me know if there is anything else I can provide or if there is anything I can try.

Thanks,

dboehlke commented 8 years ago

As an experiment, I switched nProbe's configuration to tell it to use V9 flow records instead of V10 (using -V=9 rather than -V=10 in the .conf file). The 1969 and 1970's time are gone from the host records in ntopng's display. I see that the ZMQ communications between nProbe and ntopng now has the times in fields 21 and 22 rather than 151 and 152.

(flowb{"8":"216.17.8.7","12":"38.81.66.167","15":"0.0.0.0","10":517,"14":552,"2":5000,"1":280000,"22":1447106683,"21":1447106683,"7":4283,"11":49668,"6":24,"4":6,"5":0,"16":62715,"17":174,"9":0,"13":0,"42":74574}
------ Write 251B to   10.60.59.15:56407->10.60.59.14:freeciv (nprobe)

(flowb{"8":"216.17.8.176","12":"38.81.66.94","15":"0.0.0.0","10":521,"14":552,"2":5000,"1":665000,"22":1447106683,"21":1447106683,"7":4283,"11":58546,"6":24,"4":6,"5":0,"16":62715,"17":174,"9":0,"13":0,"42":74575}
------ Write 251B to   10.60.59.15:56407->10.60.59.14:freeciv (nprobe)

(flowb{"8":"216.17.8.7","12":"38.81.66.155","15":"0.0.0.0","10":517,"14":552,"2":10000,"1":560000,"22":1447106670,"21":1447106683,"7":4283,"11":53073,"6":24,"4":6,"5":0,"16":62715,"17":174,"9":0,"13":0,"42":74576}
------ Write 256B to   10.60.59.15:56407->10.60.59.14:freeciv (nprobe)

I am checking now to see if ElasticSearch is getting good exports.

lmangani commented 8 years ago

Thanks for the update - The data exported to Elasticsearch should be as displayed in ntopng

dboehlke commented 8 years ago

I have been watching the exports since I made the change and was just using Kibana4 to look at the data and the dates all appear correct now.

dboehlke commented 8 years ago

And the activity map works in the host record now too!

lmangani commented 8 years ago

Resolved, closing.