Data sets not forwarded to storage plugin if previous data set features no template

[rickhofstede]

While developing and debugging intermediate plugins for IPFIXcol, I observed the following behaviour. While looping over the data sets of a particular IPFIX message, I found out that one of the data sets had no template associated to it:

for (i = 0; i < 1024 && msg->data_couple[i].data_set; ++i) {
    templ = msg->data_couple[i].data_template;
    if (!new_templ) {
        // No template associated to data set
    }
}

By verifying the used trace, I realised that this was not a problem of IPFIXcol in general or a previous intermediate plugin, but an artifact in the trace. The IPFIX message referred to in this ticket features four datasets, set0 - set-3, and set-2 is the data set with no template associated to it. Set-3 is however perfectly fine. The problem is now that, without any hacks, set-3 is never forwarded to storage plugins. This seems to be a generic problem related to data sets with a higher set ID than the set with no template associated to it.

Datasets can be provided upon request.

[mikeek]

That's weird. Each intermediate and storage plugin checks data_template in data_couple. If it's NULL, data set is skipped and processing moves to next one. Can you please give us a list of active plugins (both intermediate and storage) when this happens?

[rickhofstede]

Although I'd love to provide you with a list of plugins that trigger this specific behavior, I simply can't at this moment. Perhaps de odip and joinflows plugins would, since they also prepare new IPFIX messages. I haven't tested this yet though.

Let me explain a bit better about the specific behaviour I observe. So I prepare a new IPFIX message and just before I pass it back to the framework again (using pass_message) I print the message length: 1292 bytes. This is also the length that is written to the new message's header (length field) and matches exactly the length of set-0, set-1 and set-3 referred to before, so skipping set-2. However, what I get out of IPFIXcol again ('exported' using the forwarding plugin) is a message of 1116 bytes, which merely includes set-0 and set-1. And the strange thing is: even the IPFIX message header length field reports 1116 bytes, although I set it to 1292 bytes before. Maybe Wireshark tries to be smarter than it should here. I'll have to check the code of the IPFIX dissector these days.

And now comes the magic: when I manually copy the data_couple of set-3 to set-2 and set set-3 to NULL, the reported message is 1292 bytes. I know that this is not intended to work, but I use it just as a 'hack' to see what happens.

Perhaps there's still some problem with my plugin, but I wonder whether the described behaviour rings a bell for you. By the way, I noticed your commit 066acafba38e2c1feb4d241292593d56536fafbe yesterday. I haven't tried it yet but it sounds like you have/had a suspicion on what could be going wrong, right?

[mikeek]

I found that data_couple in new ipfix messages in joinflows and odip plugins was indexed wrong. Both old and new messages used the same index i. But when some data_template was missing, gap in new message was created (data_couple[i] == NULL). I added another index for new ipfix message and everything seems to be ok.

Thanks for the report