search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.03k stars 3.8k forks source link

ccl/storageccl/engineccl: enable WAL recycling with encryption at rest #42460

Open petermattis opened 4 years ago

petermattis commented 4 years ago

WAL recycling is an important optimization within Pebble/RocksDB as it reduces the number of disk sync operations required to sync the WAL file by eliminating any need to sync the metadata in the inode (WAL preallocation appears to the accomplish the same on the surface, but actually doesn't on most filesystems).

WAL recycling is currently incompatible with encryption at rest because the logic to replay WAL files at startup does not like non-empty WAL files with zero readable entries. Rather than attempting to change that logic in Pebble (and diverge for RocksDB), we could instead avoid creating non-empty WAL files during recycling. encryptedFS.ReuseForWrite could mark the encryptedFile such that the first Sync of the encrypted file causes it to be renamed from the old name to the new name. In effect, the first write to the file would occur under the old WAL name which is safe as Pebble will not be replaying the log with that name at startup. Once the sync is successful the file would be renamed (and we'd need to sync the WAL directory).

This approach seems feasible, but it might be easier to just relax the restriction on non-empty WALs with zero readable entries.

Jira issue: CRDB-5355

Epic CRDB-16419

github-actions[bot] commented 3 years ago

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!

jbowens commented 3 years ago

Just so we have a record of this, if we implement this we can change the semantics of ReuseForWrite and drop this stat that I introduced during WAL rotation.

https://github.com/cockroachdb/pebble/blob/cbda11b8689ff2093da5868a84ff00b54120d21c/db.go#L1496-L1508

github-actions[bot] commented 10 months ago

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!