search

BBVA / qed

The scalable, auditable and high-performance tamper-evident log project
https://qed.readthedocs.io/
Apache License 2.0
96 stars 19 forks source link

Reduce the size of raft snapshots by only saving the last value of each key #57

Closed panchoh closed 6 years ago

panchoh commented 6 years ago

Following @aalda's indications, I've ported the method Backup() from the badger sources to our code, tweaking the iterator options so that only the last value of each key is saved.

The result of this change is an reduction of 25% in the size of the raft snapshots (test done by adding 1M elements to qed).

codecov-io commented 6 years ago

Codecov Report

Merging #57 into master will decrease coverage by 0.97%. The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #57      +/-   ##
==========================================
- Coverage   70.45%   69.47%   -0.98%     
==========================================
  Files          39       39              
  Lines        1990     2018      +28     
==========================================
  Hits         1402     1402              
- Misses        465      493      +28     
  Partials      123      123
Impacted Files Coverage Δ
storage/badger/badger_store.go 65.13% <0%> (-14.71%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ee53bb9...16baedc. Read the comment docs.