phaistos-networks / TANK

A very high performance distributed log service
Apache License 2.0
938 stars 70 forks source link

Gotcha: Multiplexed consumed messages access, etc. #46

Open markpapadakis opened 7 years ago

markpapadakis commented 7 years ago

The Tank client, for performance, will use an internal to a connection buffer for the consumed messages. That is, instead of allocating memory to hold each consumed message, it simply points to that message in the connection buffer. This provides great performance benefits. There is catch however that need to be accounted for by the application developer:

When you execute another Tank operation, while you are iterating consumed messages or otherwise accessing those, that connection buffer is reset or freed, so those messages cannot be accessed(invalid memory access). So you should either copy them somewhere yourself, or execute those new Tank calls when you no longer need to access the original messages set.

We should simplify this by creating a new utility method that accepts an std::vector<partition_content> & and a simple_allocator and allocate memory from that allocator and update the pointers to point to that instead.