Binary encoding should be stateless

[romaindoumenc]

We have seen multiple instances where the flow between SNORT and the SecurityHub consumer would have trouble synchronizing (typically a process restart).

One key lever we can pull to make the process more robust is to ensure that we keep stateful entries small, giving the parser the option to re synchronize.

What I’m suggesting to do:

• A little buffering is OK within the logger, but every write to the pipe should start a new BILL message (even if the message contains more than one record)
• The dictionary encoding is elided.

[mike-at-trout-software]

Is the idea to resync by scanning for the "BILLxx" prefix if something is detected as being out of sync?

[romaindoumenc]

Yes, that’s the case for the scanner (I think the prefix is unique enough that we can use it). For the logger, that mean that we need to put enough BILLs to allow for the synchronisation.

PS: still chuckling at saying BILL loud all the time.

[mike-at-trout-software]

We might want to add a checksum to the end of each bill record as part of the evolution of the communication format.

I would really like to understand the details of the cause of the failure we are currently seeing, my worry is that even the current proposal might seem to fix the issue, there could be an underlying issue we are overlooking.