search

couchbase / moss

moss - a simple, fast, ordered, persistable, key-val storage library for golang
Other
959 stars 58 forks source link

add checksums #29

Open steveyen opened 7 years ago

steveyen commented 7 years ago

@hisundar had an idea / discussion on ckecksums...

Every 4K written in either the kvs array or the buf array as part of segment can be checksummed. The checksums can be stored in a new system child collection.

A new ReadOption might allow the user to either ask for optional checksum check (or allow folks to skip checksum checking) as the segment is accessed. We know mmap will pagefault in entire pages (like 4k), so after the page fault, access of 4k to double-check the checksum should be cheap (in RAM, but not necessarily in cacheline).

steveyen commented 7 years ago

Of note, the checksums would only work at the base layer (written by the persister), not for the dirty triangles that are memory-only.