From my analysis (see #10) it looks like the main reason for our massive perofrmance degradation is that we use HashMap<String, Object> for the transport of values. But we have too many operations for each record which adds up and leads to several Millions of operations for a mediocre sized batch of messages (140k messages lead to 80 Mio hash calculations on String).
So I suggest to rewrite the value handling internally (not externally) and this should lead to a massive performance gain (I expect x10 - x100 speedup).
There are different ideas I have and I'll try to document it.
But I wanted to open an Issue here to track work on this topic.
From my analysis (see #10) it looks like the main reason for our massive perofrmance degradation is that we use HashMap<String, Object> for the transport of values. But we have too many operations for each record which adds up and leads to several Millions of operations for a mediocre sized batch of messages (140k messages lead to 80 Mio hash calculations on String).
So I suggest to rewrite the value handling internally (not externally) and this should lead to a massive performance gain (I expect x10 - x100 speedup). There are different ideas I have and I'll try to document it.
But I wanted to open an Issue here to track work on this topic.