Reuse buffers on copying values

[slav]

As it was mentioned in other tickets it's not great that data is being copied on every read. In addition to that the byte array is always allocated.

It would be nice if support was added to reuse buffers somehow, so that when not needed the buffer would be reused next time to eliminate frequent allocation / GC.

One option is to use https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream and return mem stream. When stream is disposed, buffer is returned to the pool. This would lead to adding methods to return stream in addition to byte array.

[CoreyKaylor]

Happy to take a PR for this.

[abdullin]

Using RecyclableMemoryStream requires plugging your own instance of the memory manager (ideally you have only one per process), which would be rather invasive for the library.

I wonder, if it is possible to do something similar to go bindings for LMDB. There you can set ReadRaw flag on the transaction and get raw access to the unmanaged memory. Then it will be your responsibility to deal with it and remember that it will be available only while the tx is running.

[lbergnehr]

@abdullin, I'm guessing that would be something along the lines on UnmanagedMemoryStream, mentioned in #83? In terms of low memory pressure, I have a hard time imagining anything better. But as @CoreyKaylor mentioned, it might be worse in terms of read/write performance.

[abdullin]

@lbergnehr I'm afraid that UnmanagedMemoryStream might not perform well in some scenarios. Depends on the use case.

The scenario I'd love to have is to be able to get pointer to the unmanaged memory and then copy it to a buffer from the recyclable memory pool using my own unsafe methods.

[Horusiath]

Maybe instead of providing some higher level abstractions, just allow Put \ Get methods to operate on buffer: byte[], startIndex: int, length: int? A lot of abstractions (streams, array segments) could potentially work on top of it.

[CoreyKaylor]

This might be useful in the near future. https://github.com/dotnet/corefx/issues/18946

[abdullin]

Corey - just a side question. Do you use LMDB on Linux with .NET Core? How does it work and how is the performance?

On Wed, Apr 26, 2017 at 2:11 AM, Corey Kaylor notifications@github.com wrote:

This might be useful in the near future. dotnet/corefx#18946 https://github.com/dotnet/corefx/issues/18946

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CoreyKaylor/Lightning.NET/issues/85#issuecomment-297166184, or mute the thread https://github.com/notifications/unsubscribe-auth/AAezznwxukBrshXcz33Nzmixlr8__5MWks5rzmFvgaJpZM4LEQnX .

-- Rinat Abdullin | Software Engineer | Writer at Abdullin.com http://abdullin.com/

[CoreyKaylor]

@abdullin It definitely works. I'm not using it in production on Linux, but my unscientific profiling has shown slightly better performance than Windows. I've tended to compile the Linux liblmdb.so from source as you can see for the build.sh file for travis-ci.

[MaximGurschi]

How about as a start introducing an UnsafeCursor that allows putting/getting {MDB_val, MDB_val} pairs? It looks like 3x speedups are achievable (a lot more if you are reading sections) if only the under the hood ValueStructures were exposed. Then the caller can decide how to wrap that behind whatever safe mechanism they chose (Span, Memory, etc). If such a cursor were to be provided would you consider including it?

[CoreyKaylor]

While the repo has definitely been neglected for some time, my circumstances have changed and I'll be back to making some updates. I have a branch updating to the latest netstandard changes and will plan to use Span api to gain some performance benefits.

[CoreyKaylor]

Closing this issue in favor of https://github.com/CoreyKaylor/Lightning.NET/issues/110