search

CESNET / ipfixcol

IPFIXcol is an implementation of an IPFIX (RFC 7011) collector
Other
64 stars 37 forks source link

Is there a way to control file buffering when writing to json files? #192

Closed halaney closed 7 years ago

halaney commented 7 years ago

I am trying to use ifpixcol to automate some tests of my IPFIX exporter and am having some troubles with buffering (I've specifically been using the json storage option). It seems the data buffers for sometime before writing and I was wondering if this is configurable? I've tried using the dump interval but that has just caused multiple files to be waiting for the buffer to be cleared.

halaney commented 7 years ago

Also is there a way to save off the template ID for the data record? I cannot find a way to configure it to do that

thorgrin commented 7 years ago

I'm not aware of any buffering in the JSON plugin, especially when writing to files. We just use simple fwrite (https://github.com/CESNET/ipfixcol/blob/master/plugins/storage/json/File.cpp#L268) to write data to file. Perhaps the OS does the buffering? When you terminate IPFIXcol process, the data should be flushed anyway.

The dump interval is for splitting the output to multiple files, which is useful for long data reception.

There is no option for saving template ID at this time. Since the JSON structure is key:value, the template ID does not matter for applications reading the output. If you need to have it there, you might consider adding a new element with template ID to your records in the exporter. Another option would be to create an intermediate plugin which would manipulate the records and add the new element in the collector, but I guess that you would not want to do that, if you are testing the output of the exporter. It would also be possible to just write it in the JSON plugin (directly after https://github.com/CESNET/ipfixcol/blob/master/plugins/storage/json/Storage.cpp#L197), but I'm reluctant to do that since it seems to be a very specific use case and would only increase the message size for all other use cases.