Handle More IPFIX Timestamp Schemes

[Zalgo2462]

Some IPFIX exporters give timestamps in relation to the initialization time of the exporter.

We use absolute timestamps to divide out the data into separate databases and provide useful information in RITA.

More discussion is needed.

[Zalgo2462]

Definition of FlowStartSysUpTime:

The relative timestamp of the first packet of this Flow. It indicates the number of milliseconds since the last (re-)initialization of the IPFIX Device (sysUpTime). sysUpTime can be calculated from systemInitTimeMilliseconds.

https://www.iana.org/assignments/ipfix/ipfix.xhtml

[Zalgo2462]

https://tools.ietf.org/html/rfc5153#section-4.7

Well... ipfix records SHOULD include the systemInitTimeMilliseconds field with each flow.... but its not a requirement

[Zalgo2462]

A piece of software from cisco requires systeminittimemilliseconds be present when processing relative timestamps in ipfix.

https://www.cisco.com/c/en/us/td/docs/net_mgmt/prime/performance/1-6/user/guide/PrimePerformanceManager-1-6-UserGuide/ppmsettingupreports.html

Valid IPFIX combinations include:

    systemInitTimeMilliseconds, flowStartSysUpTime, flowEndSysUpTime

Note If start and end uptime fields are sent in the flow, the systemInitTimeMilliseconds field is also required to calculate the reporting interval.

    flowStartSeconds, flowEndSeconds
    flowStartMilliseconds, flowEndMilliseconds
    flowStartMicroseconds, flowEndMicroseconds
    flowStartNanoseconds, flowEndNanoseconds
    flowStartDeltaMicroseconds, flowEndDeltaMicroseconds

[Zalgo2462]

Archiving until data is available to test the different timestamp combinations

[Zalgo2462]

SonicWALL and Mikrotik seem to send flows with flowStartSysUpTime and flowEndSysUpTime, but they never send systemInitTimeMilliseconds.

These devices do support Netflow v9 however, and the v9 standard requires sending the uptime along with relative timestamps.

[Zalgo2462]

40 Details an implementation of IPFIX which sends systemInitTimeMilliseconds along with flowStartSysUpTime nad flowEndSysUpTime.

[Zalgo2462]

@SamuelCarroll suggested a method of recovering the flowStartSysUpTime. It may be possible to calculate the flowStartSysUpTime by subtracting the flowEndSysUpTime from the timestamp at which Logstash received the flow. For IPFIX devices which continuously export their flows in real time, this approach should be roughly accurate, although it will become less accurate during periods of intense traffic. For devices which send flows in batches, this approach will not work.

[SamuelCarroll]

@william-stearns had a really good idea, since RITA is primarily interested in time differences we can use the first flow of a day and get two values from it. The first is it's measurement of milliseconds since system init time (msBase) and then use the logstash @timestamp (call this localTS). Then for every flow that comes in for the rest of the day take it's start (flowStartSysUpTime) and end (flowEndSysUpTime) to find the time delta from when it originally arrived. So for any flow it's start time is localTS + (flowStartSysUpTime - msBase) and it's end time is `localTS + (flowEndSysUpTime - msBase)

The advantage to this approach is times may be off of their true time by some unknown constant but they should all be relative to one another and the time delta's are correct, and get accurate results. I can work through the math later but I think this would be the best approach