cojen / TuplDB

TuplDB is a high-performance, concurrent, transactional, scalable, low-level embedded database.
GNU Affero General Public License v3.0
110 stars 22 forks source link

Log compaction during restore #80

Closed broneill closed 5 years ago

broneill commented 6 years ago

During a restore operation, the receiving member collects segments for all the data it needs to apply after the snapshot has been received. Because no data is applied during the restore, no calls to syncCommit are made. This in turn forces all members of the group to retain segment files during the restore, even when they're calling compact. During the restore, calls to syncCommit should be made automatically, up to the current commit index, perhaps once a second.

After the restore, the peer begins applying changes, but it might take a long time to catch up. This also prevents the other members from compacting, and so perhaps syncCommit should always be called automatically, against the current commit index. Typically the application can only call syncCommit against the highest applied index.