search

komora-io / marble

garbage-collecting on-disk object store, supporting higher level KV stores and databases.
Apache License 2.0
352 stars 12 forks source link

`Marble::read` buffer alignment #1

Open bonsairobo opened 2 years ago

bonsairobo commented 2 years ago

I think it would be useful to guarantee some minimum alignment of the buffer returned from Marble::read, which would allow the buffer to be used with zero-copy deserialization like rkyv. I know that sled aligns IVecs to 16 bytes (on the main branch). I hacked this into my fork of marble here.

spacejam commented 2 years ago

Hey @bonsairobo I'm thinking about allowing Marble::read to take a generic allocator as an argument, in the same way that the std::collections do, and in your allocator you can make it a simple pass-through that calls max on the alignment with your desired minimum alignment before passing it to the default allocator.

bonsairobo commented 2 years ago

@spacejam That sounds good. Although if the container I want to use (e.g. rkyv::AlignedVec) is not generic over allocators, then it is hard to use this API. Maybe you'd rather like a trait similar to the Buffer trait I've got here.

fabianmurariu commented 1 year ago

@spacejam any chance you would revisit this? I'm also trying to use rkyv with marble (well with any storage layer for that matter)

rdaum commented 1 year ago

I guess it would help if I had read this thread first before filing my other issue. Some duplication of conversation here.

FWIW I vote highly for this approach.