search

CESNET / ipfixcol

IPFIXcol is an implementation of an IPFIX (RFC 7011) collector
Other
64 stars 37 forks source link

FastBit storage plugin: meaning of renamed table directories? #18

Closed ghost closed 9 years ago

ghost commented 9 years ago

In one of our deployment setups, where we receive flow data from a single INVEA-TECH FlowMon Probe over IPFIX per IPFIXcol instance with FastBit storage, we notice that table directories (i.e., those featuring template IDs) are renamed quite regularly. This results in table directories with alphabetic postfixes, such as 'a' and 'b'. I'm wondering what is exactly the reason for those postfixes.

While checking the code, I found out that the dir_check routine in plugins/storage/fastbit/fastbit_table.cpp is where the renaming is done. This routine adds a new directory with the mentioned postfixes if the (original) directory already exists. So now I'm wondering: what is the meaning of the renamed directories? Why isn't the data simply merged? Is it an indication of an issue, since I'm receiving data from just a single ODID?

thorgrin commented 9 years ago

Basically, using UDP, the templates are exported repeatedly. However, if the template changes and retains the same number, we must take this in to account in the storage, since the table structure has changed. Therefore, we simply add prefix if the template was updated.

However, we currently assume that the template always changes which causes new suffix to be create each time the template is exported again, which in case of default flowmonexp settings is quite often. We are working on an update to check whether the template really changed and provide the storage plugins with this information.

thorgrin commented 9 years ago

The desired behavior is now implemented.