search

CESNET / ipfixcol

IPFIXcol is an implementation of an IPFIX (RFC 7011) collector
Other
64 stars 37 forks source link

Storage plugin initialization on startup #125

Closed ghost closed 9 years ago

ghost commented 9 years ago

The current implementation initializes storage plugins only in the moment in which the plugin is actually needed, i.e., once there's something to be stored. If initialization fails though, IPFIXcol just continues operation. This behavior is extremely error-prone, especially in unattended setups. I therefore propose the following changes:

thorgrin commented 9 years ago

I agree that late initialization can be a problem. However, this is quite integral part of the ipfixcol now. We have an input plugin and a number of intermediate plugins in a queue after that. From there, things are a bit more complicated. The last intermediate plugin passes data to output manager, which checks ODID of the source and passes the data to an ODID specific data manager. The data manager then hands the data over to the configured storage plugins. However, for every ODID there is new, dynamically allocated data manager, which initializes its own plugins. Thus, without knowing all used ODIDs beforehand, we are not able to init the data manager and its storage plugins at the startup time.

To sum up, only output manager can be initialized on startup, the rest is dynamic. We could probably add some configuration to use only one data manager and have it handle all ODIDs, then it can be initialized on startup. However, that is a significant change to current behaviour, although there might be use cases for that.

ghost commented 9 years ago

Is it right that 7c26062bededc3a81a242cfc30625cdb7ae2b051 is meant to make the data manager handle all ODIDs, as discussed in the previous comment? I believe it would be good to document this kind of options, as it may be useful for others as well and is currently hidden in the list of commits.

thorgrin commented 9 years ago

Yes, it is. I suggest that we add the configuration option to the example startup.xml. It should also be mentioned at https://github.com/CESNET/ipfixcol/tree/devel/base since it shows the architecture schema which is affected by this change. Do you think that is enough?

ghost commented 9 years ago

That would be a big help already, indeed. A concrete use case could also be added, perhaps. Maybe something along the lines of "if you don't need to have different storage procedures/plugins for every ODID, use this".

Lukas955 commented 9 years ago

In commit 475939faa902ce16dce677e01bd5b1bfbe503456 I added a documentation of single Data manager into base/README.md and startup.xml. Do you think that is enough?

ghost commented 9 years ago

We will soon start testing this functionality. Please consider the current state to be 'enough' for now. In case we encounter some unclarities, I'll come back to you. Thanks!