gchq / stroom-stats

Other
4 stars 4 forks source link

Use a faster serialisation format for the stat events #53

Open at055612 opened 5 years ago

at055612 commented 5 years ago

On profiling, a lot of time is spend unmarshalling the xml. If stroom were to put stat events on kafka in something like flatuffers then the serde cost could be reduced. There also appears to be a noteable cost in converting the datetime in the xml into a java instant. The downside to flatbuffers is the added complexity, inability to human read the messages on the topic.

at055612 commented 5 years ago

An alternative would be to use the dsl-json (de)serialiser (https://github.com/ngs-doo/dsl-json). This claims to be on par with binary formats for performance but is human readable. This would also make it easier for other things to generate stat events as it is just plain old json.