ysono / pancake

7 stars 1 forks source link

Range query. #50

Closed ysono closed 3 years ago

ysono commented 3 years ago

Note, we are using KeyValueIterator for all operations that merges across multiple SSTables. This is inefficient, as we are unnecessarily deserializing values. This can be addressed later.

I'm not sure if the native + itertools abstractions of iterators can handle this use pattern. I actually tried using some ugly raw pointer passing to try to give access to the iterator so that you could call iter.get_value() on the iterator. This is a bad idea! Seg fault occurs. It shows that rust's ownership transfer does really cause a c++ std::move() kind of moving.

btc commented 3 years ago

Note, we are using KeyValueIterator for all operations that merges across multiple SSTables. This is inefficient, as we are unnecessarily deserializing values. This can be addressed later.

It's worth benchmarking to determine the cost associated with deserialization. With respect to inducing latency, it might be tiny compared to other costs (such as disk accesses). With respect to memory usage, it might be small since there's theoretically only a constant size allocation. With respect to CPU cycles, it might be significant. But worth measuring when contemplating decisions that treat the cost as a material constraint which ought to influence code organization and system architecture.

ysono commented 3 years ago

:+1: