search

tikv / rust-rocksdb

rust wrapper for rocksdb
Apache License 2.0
276 stars 155 forks source link

Support for ChecksumGeneratorFactory #739

Open arcmarqs opened 1 year ago

arcmarqs commented 1 year ago

It would be nice to have a trait for implementing customized checksum generator factories as described in https://github.com/facebook/rocksdb/wiki/Full-File-Checksum-and-Checksum-Handoff#how-to-use-full-file-checksum

The API is here https://github.com/facebook/rocksdb/blob/main/include/rocksdb/file_checksum.h

I'm not sure how this could be implemented but I would be glad to help.

tabokie commented 1 year ago

Yes it would be nice. But before we start working on the details, I want to be sure that you understand the current status of this crate. Right now, the master branch is linked with TiKV's fork of RocksDB 6.29 (https://github.com/tikv/rocksdb). And we don't have a clear plan to update it yet. In fact, we are lowkey working on a new binding crate tirocks designed to replace this one. When that happens, this crate will likely be archived.

arcmarqs commented 1 year ago

The version isn't a problem as 6.29 supports full file checksum. But in that case, do you recomend using tirocks instead?

tabokie commented 1 year ago

do you recomend using tirocks instead?

tirocks is not production ready yet. But it would be fairly easy to port full checksum then because tirocks has a superior binding generation workflow.

The version isn't a problem as 6.29 supports full file checksum.

Great. To port full checksum in rust-rocksdb, you can start with the existing code of compaction filter. They have a very similar structure.

My recent commit refactored the compaction filter interface (https://github.com/tikv/rust-rocksdb/commit/9e4678857e5b4c738e95c7ee1a35ee962264f4e9). You can pretty much see how a binding is implemented: (1) declare ffi in c.h (2) implement ffi in c.cc (3) implement rust wrapper of raw C pointer. For structures involving callbacks (such as compaction filter or checksum generator), we need to pass Rust function as C function pointer, and Rust struct as C raw pointer. You can see it in action with CompactionFilterContext or CompactionFilterFactoryProxy.

arcmarqs commented 1 year ago

Thanks for the directions! I'll look into it.