search

CESNET / ipfixcol2

High-performance NetFlow v5/v9 and IPFIX collector (RFC7011)
Other
124 stars 36 forks source link

CSV output format #71

Open YATV opened 2 years ago

YATV commented 2 years ago

Hello! If you save it to a file in the JSON format, then a lot of traffic is generated and it takes up a lot of space (even with compression). It would be very nice to be able to save to a file in CSV format Thank you!

Lukas955 commented 2 years ago

Hi,

to be honest, I don't think any text file format is suitable for storing big data.

The disadvantage of CSV formatting is that you need to directly specify the fields that should appear in each column. In other words, you need to know exactly what is in the data of the exporter and what you want to store. In contrast, the good thing about JSON is that it is a very flexible format that allows you to easily convert very flexible IPFIX records.

We currently have no plans to add support for formatting to a CSV file in the near future.

Lukas

sedmicha commented 2 years ago

What you can currently do is store the data in the Nffile format and then use a tool like Nfdump to read the data as CSV. There's a problem that Nfdump doesn't fully support all that IPFIX has to offer, but if you don't use any special fields it might be a suitable solution for you.

There's also the FDS format designed to be even more efficient, but the equivalent Fdsdump tool is still in development. Once that's done, that would probably be the better choice.