search

n-young / trustdb

0 stars 1 forks source link

Maintain unique start_timestamps for indexing #16

Open desmondcheongzx opened 3 years ago

desmondcheongzx commented 3 years ago

In #14 we introduce a B-tree block index mapping from start_timestamp to filename. There's currently nothing enforcing unique start_timestamps besides an expectation of well-behaved data. We can consider creating a sort of "timestamp checkpoint", where we refuse to keep any data with timestamps less than or equal to our latest start_timestamp.

If we want to avoid losing late data, we can consider keeping the block around for awhile in-between resetting the active block and flushing the old block to disk. Something like Timon.

n-young commented 3 years ago

We now map from start_timestamp => vec, I think this fixes it? Close it if you think so

n-young commented 3 years ago

New issue: timestamps don't even come remotely in order