CoreyKaylor / Lightning.NET

.NET library for LMDB key-value store
Other
397 stars 82 forks source link

Support ReadOnlySpan<byte> where byte[] parameters are used #124

Closed danielcrenna closed 4 years ago

danielcrenna commented 4 years ago

Methods like cursor.MoveTo(byte[] key) and similar all call private methods that cast the byte[] parameter to a ReadOnlySpan<byte>.

Can we get overloads (or replacements) that take the ReadOnlySpan<byte> directly?

This would save unnecessary allocations.

AlgorithmsAreCool commented 4 years ago

Yes, we are looking at updating the cursor API, we should have something "soon™".

Just curious, what kind of keys do you generally use? Integers? Strings? blobs?

danielcrenna commented 4 years ago

Almost always it's a composite string key prepared with Encoding.UTF8.GetBytes because it's easier to build keys for lexicographical streaming that way, but I'd like to reduce these allocations, even an LRU cache on this still means I'm holding a large bag of byte buffers.

AlgorithmsAreCool commented 4 years ago

Yep, we should be able to quash those allocations easily.

Also, I'm assuming you mean keeping keys implicitly sorted in lexicographic order. Maybe we can optimize for this use case. A little stackalloc goes along way

CoreyKaylor commented 4 years ago

I have a bunch of the changes made locally. I'll push up a PR in the next day or two. Had a little bit of a tangent when I thought I found a bug in the underlying native library, but select isn't broken so I should be able to move forward again.

danielcrenna commented 4 years ago

@AlgorithmsAreCool yeah, so what I was intending to do was to store prefixes and key segments in contiguous ranges, so that I could produce a prefixed key without having to allocate memory. I've done this just with a fake "RangeSpan" type of construct for the time being, which does length counting and switches spans based on the index.

CoreyKaylor commented 4 years ago

Any suggestions for the PutMultiple overload?

danielcrenna commented 4 years ago

@CoreyKaylor thanks for this, API looks really solid. cursors took more time to refactor but it's better off.

CoreyKaylor commented 4 years ago

@danielcrenna thanks for the comment. Always nice to hear some positive feedback.