json file writer flowStartSeconds and flowEndSeconds seem to be bigger values than possible

[jeremyforan]

I am outputting using the json file writer and the timestamps cant possibly be correct

{
    "@type": "ipfix.entry",
    "destinationTransportPort": 53,
    "flowEndSeconds": 6624567418595311616,
    "flowStartSeconds": 6624567332695965696,
    ...

<?xml version="1.0" encoding="UTF-8"?>
<ipfix xmlns="urn:ietf:params:xml:ns:yang:ietf-ipfix-psamp">
    <collectingProcess>
        <name>UDP collector</name>
        <udpCollector>
            <name>Listening port 4739</name>
            <localPort>4739</localPort>
            <templateLifeTime>1800</templateLifeTime>
            <optionsTemplateLifeTime>1800</optionsTemplateLifeTime>
            <localIPAddress></localIPAddress>
        </udpCollector>
        <exportingProcess>File writer</exportingProcess>
    </collectingProcess>

    <exportingProcess>
        <name>File writer</name>
        <destination>           
            <name>JSON storage plugin</name>
            <fileWriter>
                <fileFormat>json</fileFormat>

                <tcpFlags>formated</tcpFlags>
                <timestamp>unix</timestamp>
                <protocol>formated</protocol>
                <ignoreUnknown>yes</ignoreUnknown>
                <nonPrintableChar>no</nonPrintableChar>
            <prefix></prefix>               
        <output>
            <type>file</type>
            <path>/tmp/ipfixcol/flow/%Y/%m/%d/</path>
            <prefix>json.</prefix>
            <dumpInterval>
                <timeWindow>300</timeWindow>
                <timeAlignment>yes</timeAlignment>
            </dumpInterval>
        </output>
            </fileWriter>
        </destination>
        <singleManager>yes</singleManager>
    </exportingProcess>
<!-- List of active Intermediate Plugins -->
    <intermediatePlugins>
    </intermediatePlugins>
</ipfix>

[thorgrin]

What is your flow data source? Could you look at the flow data in Wireshark to check what timestamps are in the flow data? If not, can you provide sample of the flow data (a pcap perhaps)?

We have been using this plugin quite a lot so I guess there is some misconfiguration or misinterpretation of the source data, such as microseconds being interpreted as milliseconds. Or maybe, in case of NetFlow v9, there may be a problem with the conversion to IPFIX. But we need a bit more information to find the cause.

[jeremyforan]

I have the pcap open in wireshark, not sure how to hone in on the flowStartSeconds value. Is there a reference you can point me to?

[thorgrin]

Maybe this tutorial will help: https://helpdesk.kaseya.com/hc/en-gb/articles/115003522631-How-to-view-NetFlow-in-WireShark

You just need to find your NetFlow packet (using cflow in the filter field) and then open the payload. Any packet with data sets should have the timestamp if you open individual flows.

[jeremyforan]

As fair as I can tell everything looks fine for the IPFIX data I am receiving.

tcpdump -i any "port 4739" -w /tmp/ipfix.pcap screenshot

[thorgrin]

It seems that you are receiving the timestamps in dateTimeSeconds format. There was definitely a bug in the code handling this format. I've fixed it in devel branch. Could you rebuild the json plugin from devel and try it again? Unfortunately, I do not have any source which would send timestamps as seconds, so I cannot easily test it myself. If you need an rpm package, let me know, I can create one for your.

[jeremyforan]

an RPM would be ideal.

Sent from my iPhone

On Nov 19, 2018, at 08:26, Petr Velan notifications@github.com wrote:

It seems that you are receiving the timestamps in dateTimeSeconds format. There was definitely a bug in the code handling this format. I've fixed it in devel branch. Could you rebuild the json plugin from devel and try it again? Unfortunately, I do not have any source which would send timestamps as seconds, so I cannot easily test it myself. If you need an rpm package, let me know, I can create one for your.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[thorgrin]

I've created a RPM for Centos 7 system. You just need to reinstall it locally. If everything is fine, I'll increase version and push it to the package mirror. json.zip

[jeremyforan]

I am running AWS EC2 Linux 2, it didn't like the rpm https://aws.amazon.com/amazon-linux-2/

[root@tmp]# yum install ipfixcol-json-output-1.2.5-1.src.rpm 
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Examining ipfixcol-json-output-1.2.5-1.src.rpm: ipfixcol-json-output-1.2.5-1.src
Cannot add package ipfixcol-json-output-1.2.5-1.src.rpm to transaction. Not a compatible architecture: src
Error: Nothing to do

[thorgrin]

Sorry, my mistake, I uploaded the source RPM instead of the compiled binary. Please try this one. json.zip

[jeremyforan]

Its looking good!!!

[root@]# head -n1 json.201811191820 | python -mjson.tool
{
    "@type": "ipfix.entry",
    "flowEndSeconds": 1542651894,
    "flowStartSeconds": 1542651882,
     ...
}

[thorgrin]

Good, thank you for reporting and testing this. I've just created new json plugin package in the COPR repository.

[jeremyforan]

Does this mean it is now pushed out to the general repo's? what is the version?

group_CESNET-IPFIXcol/x86_64/primary_db                                                                                                     |  30 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package ipfixcol-json-output.x86_64 0:1.2.5-1 will be updated
---> Package ipfixcol-json-output.x86_64 0:1.2.6-1 will be an update
--> Finished Dependency Resolution

[thorgrin]

Yes, version 1.2.6 is the one with the bugfix.