Sophos UTM IPFIX timestamps

[ken-adey]

Nfdump is reporting "1970-01-01 00:00:00" for "first" and "last" using raw output. I suspect it's the Sophos that is at fault here. Wireshark is displaying the right times, but in looking at the templates, "flowStartSeconds" (150) is in the templates, while "flowEndSeconds" is in only one of the templates. Is this right? In previous discussions of IPFIX timestamp issues on the nfdump mailing list and on here, these 2 tags ("flowStartSeconds" and "flowEndSeconds") aren't mentioned.

[ken-adey]

Addendum to my original submission; in looking in the code, the above mentioned tags are not supported. Could this be the problem?

[bbayles]

Do you have a PCAP that shows the issue that you can share?

[ken-adey]

I do, attached.

Thanks.

On Mon, Jul 1, 2019 at 8:13 PM Bo Bayles notifications@github.com wrote:

Do you have a PCAP that shows the issue that you can share?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/phaag/nfdump/issues/170?email_source=notifications&email_token=AHUIHYFJRNXTKF5TYIKUG63P5JQOVA5CNFSM4H4VH6W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY7G7DA#issuecomment-507408268, or mute the thread https://github.com/notifications/unsubscribe-auth/AHUIHYE3OEICJUFVESICEI3P5JQOVANCNFSM4H4VH6WQ .

-- Ken....

[bbayles]

FYI, I don't think the Github interface attaches files via e-mail.

[ken-adey]

ipfix.zip

[bbayles]

Looks like you're right - the template specifies flowStartSeconds (150) and flowEndSeconds, but those aren't available in ipfix.h or ipfix.c.

The Sophos device seems to be packing the 256 template correctly - everything is represented properly.

Below is a script that uses TShark (a recent version with JSON support) to read a capture and dump it to stdout. You may be able to modify it for your use?

---

IPFIX to table with Tshark

```python #!/usr/bin/env python3 from json import loads from subprocess import check_output FIELDS = ( 'cflow.srcaddr', 'cflow.dstaddr', 'cflow.srcport', 'cflow.dstport', 'cflow.protocol', 'cflow.octets', 'cflow.packets', 'cflow.abstimestart', 'cflow.abstimeend', ) def main(file_path, ports=None): args = ['tshark', '-2', '-r', file_path, '-T', 'json'] for field in FIELDS: args += ['-e', field] ports = ports or [] for p in ports: args.append('-d') args.append('udp.port=={},cflow'.format(p)) tshark_output = check_output(args).decode('utf-8') all_packets = loads(tshark_output) print(*FIELDS, sep='\t') for packet in all_packets: layers = packet.get('_source', {}).get('layers', {}) values = [layers.get(f, []) for f in FIELDS] for flow in zip(*values): print(*flow, sep='\t') if __name__ == '__main__': import argparse parser = argparse.ArgumentParser('Poll AWS Kinesis') parser.add_argument( 'file_path', type=str, help='The PCAP file to read' ) parser.add_argument( '--port', type=int, action='append', help='Port to interpret as CFLOW' ) args = parser.parse_args() main(args.file_path, ports=args.port) ```

[ken-adey]

Thanks for that code.

Know any source of guidance on how to add those elements to the ipfix template?

Thanks.

On Tue, Jul 2, 2019 at 4:02 PM Bo Bayles notifications@github.com wrote:

Looks like you're right - the template specifies flowStartSeconds (150) and flowEndSeconds, but those aren't available in ipfix.h or ipfix.c.

The Sophos device seems to be packing the 256 template correctly - everything is represented properly.

Below is a script that uses TShark (a recent version with JSON support) to read a capture and dump it to stdout. You may be able to modify it for your use?

IPFIX to table with Tshark

!/usr/bin/env python3from json import loadsfrom subprocess import check_output

FIELDS = ( 'cflow.srcaddr', 'cflow.dstaddr', 'cflow.srcport', 'cflow.dstport', 'cflow.protocol', 'cflow.octets', 'cflow.packets', 'cflow.abstimestart', 'cflow.abstimeend', )

def main(file_path, ports=None): args = ['tshark', '-2', '-r', file_path, '-T', 'json']

for field in FIELDS:
    args += ['-e', field]

ports = ports or []
for p in ports:
    args.append('-d')
    args.append('udp.port=={},cflow'.format(p))

tshark_output = check_output(args).decode('utf-8')
all_packets = loads(tshark_output)

print(*FIELDS, sep='\t')
for packet in all_packets:
    layers = packet.get('_source', {}).get('layers', {})
    values = [layers.get(f, []) for f in FIELDS]
    for flow in zip(*values):
        print(*flow, sep='\t')

if name == 'main': import argparse

parser = argparse.ArgumentParser('Poll AWS Kinesis')
parser.add_argument(
    'file_path',
    type=str,
    help='The PCAP file to read'
)
parser.add_argument(
    '--port',
    type=int,
    action='append',
    help='Port to interpret as CFLOW'
)
args = parser.parse_args()
main(args.file_path, ports=args.port)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/phaag/nfdump/issues/170?email_source=notifications&email_token=AHUIHYHX5I34ENNN66EIALLP5N3YRA5CNFSM4H4VH6W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZBYHRY#issuecomment-507741127, or mute the thread https://github.com/notifications/unsubscribe-auth/AHUIHYCP5KRFWETBYLIQPC3P5N3YRANCNFSM4H4VH6WQ .

-- Ken....

[phaag]

IPFIX elements #150 and #151 added ipfix.c, ipfix.h Please check with the current code

[ken-adey]

Thanks Peter, works like a charm. I had started making the same changes, but was missing the TimeUnix piece of the puzzle.

On Tue, Jul 2, 2019 at 7:16 PM Peter Haag notifications@github.com wrote:

IPFIX elements #150 https://github.com/phaag/nfdump/issues/150 and #151 https://github.com/phaag/nfdump/issues/151 added ipfix.c, ipfix.h Please check with the current code

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/phaag/nfdump/issues/170?email_source=notifications&email_token=AHUIHYF3OAO6UFTL7MQUNSLP5OSPDA5CNFSM4H4VH6W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZCI6PQ#issuecomment-507809598, or mute the thread https://github.com/notifications/unsubscribe-auth/AHUIHYBYJB3LTBDJ5VZYRWTP5OSPDANCNFSM4H4VH6WQ .

-- Ken....

[phaag]

Great! - Just bear in mind, that you only have seconds resolution.