I've been experimenting with enabling ZSTD compression and I beehive ZSTD should be offered as an option for those who might benefit from it.
Benefits:
DB size is 15% smaller (as of Jan 2024 fully synced ZSTD compressed DB is 112GB while uncompressed one ~134GB)
Reduced I/O which might help with slower SSDs
Depending on host, sync speed shouldn't increase much if at all (on i5-6200u with 8GB RAM, sync took 29H)
I am not asking to enable it by default (as I did in the patch below) but having a config option for people to utilize would be really nice.
diff --git a/src/Storage.cpp b/src/Storage.cpp
index 8ee8a52..1001eb2 100644
--- a/src/Storage.cpp
+++ b/src/Storage.cpp
@@ -1806,7 +1806,7 @@ void Storage::startup()
opts.error_if_exists = false;
opts.max_open_files = options->db.maxOpenFiles <= 0 ? -1 : options->db.maxOpenFiles; ///< this affects memory usage see: https://github.com/facebook/rocksdb/issues/4112
opts.keep_log_file_num = options->db.keepLogFileNum;
- opts.compression = rocksdb::CompressionType::kNoCompression; // for now we test without compression. TODO: characterize what is fastest and best..
+ opts.compression = rocksdb::CompressionType::kZSTD; // for now we test without compression. TODO: characterize what is fastest and best..
opts.use_fsync = options->db.useFsync; // the false default is perfectly safe, but Jt asked for this as an option, so here it is.
shistOpts = opts; // copy what we just did (will implicitly copy over the shared table_factory and write_buffer_manager)
@@ -1834,7 +1834,7 @@ void Storage::startup()
Debug() << "DB \"" << name << "\" mem: " << QString::number(mem / 1024. / 1024., 'f', 2) << " MiB";
opts.OptimizeLevelStyleCompaction(mem);
for (auto & comp : opts.compression_per_level)
- comp = rocksdb::CompressionType::kNoCompression; // paranoia -- enforce no compression since our data compresses so poorly
+ comp = rocksdb::CompressionType::kZSTD; // paranoia -- enforce no compression since our data compresses so poorly
memTotal += mem;
rocksdb::Status s;
// try and open database
I've been experimenting with enabling ZSTD compression and I beehive ZSTD should be offered as an option for those who might benefit from it.
Benefits:
I am not asking to enable it by default (as I did in the patch below) but having a config option for people to utilize would be really nice.