komora-io / marble

garbage-collecting on-disk object store, supporting higher level KV stores and databases.
Apache License 2.0
352 stars 12 forks source link

Consider letting the user do their own buffer allocation? #13

Open rdaum opened 1 year ago

rdaum commented 1 year ago

Playing with Marble for my own fun project, and looking at the implementation of get, I was wondering if you'd consider adding an API that lets uses provide their own callback for buffer allocation?

In my case I would like to be allocating into my own bespoke buffer pool, which is based vaguely/roughly on the techniques described in the Umbra paper (https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf). And I would prefer not to have an extra copy/move step into those buffers after doing my work with Marble.

I'm imagining something like being able to pass something like an FnMut(usize)->Pin<&mut [T]> as an alloc_fn or similar? Marble still might need to read and compose the header_buf separately, but the work done with uninit_boxed_slice could be done instead by the user's provided buffer allocation routine.

Not sure if this makes sense for your overall design or not, but something to consider.