This PR highly reduces the amount of memcpy when merging multiple Readers by only using references from inside the decompressed block bytes inside the Reader. To do that we have introduced a new Reader::current method that returns the key-value pair that is currently pointed and that has been returned by the previous call to Reader::next, None if it has been called already.
The only time we copy memory is when we copy the key that is currently being merged and when the merged value is returned from the merge function in a Cow::Borrowed which means that it is borrowed from one of the arguments, it is therefore mandatory to copy it in a Vec<u8> inside of the Merger struct.
We removed all of the memcpy of the values that we want to merge, instead of storing them in a Vec of Cow<'static, [u8]> and therefore always calling value.to_vec() and storing them in a Cow::Owned. We now always store the values in a Cow::Borrowed avoiding the to_vec call.
I am not sure to understand why, but the miri CI passes now, strange but good 🤪
This PR highly reduces the amount of
memcpywhen merging multipleReaders by only using references from inside the decompressed block bytes inside theReader. To do that we have introduced a newReader::currentmethod that returns the key-value pair that is currently pointed and that has been returned by the previous call toReader::next,Noneif it has been called already.The only time we copy memory is when we copy the key that is currently being merged and when the merged value is returned from the merge function in a
Cow::Borrowedwhich means that it is borrowed from one of the arguments, it is therefore mandatory to copy it in aVec<u8>inside of theMergerstruct.We removed all of the
memcpyof the values that we want to merge, instead of storing them in aVecofCow<'static, [u8]>and therefore always callingvalue.to_vec()and storing them in aCow::Owned. We now always store the values in aCow::Borrowedavoiding theto_veccall.I am not sure to understand why, but the miri CI passes now, strange but good 🤪