search

aergoio / litetree

SQLite with Branches
MIT License
1.62k stars 35 forks source link

Pruning of old commits / revisions #16

Closed domob1812 closed 6 years ago

domob1812 commented 6 years ago

As far as I understand, litetree works by keeping copies of all versions of any changed database pages. But that also means that the total storage requirement grows over time, particularly if the database is changed very often (even if the database itself does not grow), right? Is there a way to tell litetree that some commit / old version is no longer needed and can be discarded (like checkpointing in SQLite's WAL mode)?

Particularly for blockchain applications (which seems to be the intent of litetree) this can be useful: In order to handle chain reorgs, it is necessary to keep (persistent) snapshots of the last couple of states. But as more and more blocks are added on top of an old one, it can be pruned eventually to save space. (In some sense, revisions in litetree are like undo data in Bitcoin Core, which can (optionally) be pruned for old blocks.)

Perhaps my understanding is also completely wrong or this is already supported. If so, please let me know.

kroggen commented 6 years ago

Hi Daniel,

I was researching exactly on this (merging commits) at this week.

Yes, it will be available on LiteTree. It may be released probably at this month.

domob1812 commented 6 years ago

@kroggen: Ok cool, thanks for the confirmation! I believe this can be a very powerful feature (particularly for blockchain projects as ours and yours).

kroggen commented 6 years ago

Hi Daniel!

The support for discarding old commits is available in the discard-commits branch.

If you find any bug, please report it as a new issue.

domob1812 commented 6 years ago

Thanks, good to know!