rockeet
commented
2 years ago I'm working for remote compaction of RocksDB, which is an experimental feature. When I try to use this feature in MyRocks, I encounter one problem.
It seems that the remote RocksDB instance can't initialize Rdb_tbl_prop_coll_factory because it needs m_ddl_manager, which can't be obtained in the RocksDB layer. Therefore, when calling the read_stats_from_tbl_props function, the SST file generated by the remote rocksdb instance does not contain some information required by MyRocks, resulting in a fatal error.
The MySQL error log is shown in the following figure. There is no error in RocksDB layer.
How could I hard code to initialize Rdb_tbl_prop_coll_factory in remote RocksDB instance for remote compaction without m_ddl_manager? Or are there other suggestions that can solve this problem？
This feature needs RocksDB meta objects(CompactionFilter, PropColl, ...) serialization, this is not properly supported by RocksDB. As an alternative, ToplingDB is full featured on Distributed Compaction, and we have an in house myrocks branch(on top of ToplingDB) with such features, will opensource soon.
I'm working for remote compaction of RocksDB, which is an experimental feature. When I try to use this feature in MyRocks, I encounter one problem.
It seems that the remote RocksDB instance can't initialize Rdb_tbl_prop_coll_factory because it needs m_ddl_manager, which can't be obtained in the RocksDB layer. Therefore, when calling the read_stats_from_tbl_props function, the SST file generated by the remote rocksdb instance does not contain some information required by MyRocks, resulting in a fatal error.
The MySQL error log is shown in the following figure. There is no error in RocksDB layer.
How could I hard code to initialize Rdb_tbl_prop_coll_factory in remote RocksDB instance for remote compaction without m_ddl_manager? Or are there other suggestions that can solve this problem？