curiosity-ai / rocksdb-sharp

.net bindings for the rocksdb by facebook
BSD 2-Clause "Simplified" License
155 stars 38 forks source link

Question: Is it safe to reuse buffers for batch write operation? #41

Closed Havret closed 1 year ago

Havret commented 1 year ago

I maintain RocksDb.Extensions, a .NET library that integrates RocksDB with the .NET dependency injection system. With this library, you can easily create key/value data stores backed by RocksDB, and use built-in serializer factories to serialize your data.

To reduce memory pressure, I'm striving to minimize memory allocations in the library. Specifically, I've implemented a zero allocations approach for the Put operation for a single item using array pooling and stack allocation for small payloads. Now I'm working on a similar approach for the PutRange operation. So far, my draft implementation seems to be working and all tests pass. However, I'm wondering if anyone can confirm whether this approach will be safe. My understanding is that the buffers will need to be copied during marshalling to the C++ part of the code. Can you offer any insights on this?

theolivenbaum commented 1 year ago

Hi @Havret, my understanding is that RocksDB will always copy the data you pass it so you don't have to keep it forever, so once you call

            _ = batch.Put(keySpan, valueSpan, _columnFamilyHandle);

It should be safe to re-use the arrays for something else.