twmht / python-rocksdb

Python bindings for RocksDB
BSD 3-Clause "New" or "Revised" License
274 stars 89 forks source link

How to perform manual compression in python rocksdb #87

Closed Jamiejoin closed 3 years ago

Jamiejoin commented 3 years ago

How to perform manual compression in python rocksdb

  def connect_rocksdb(self):
        opts = rocksdb.Options()
        opts.create_if_missing = True
        opts.max_background_flushes = 80
        opts.max_open_files = -1
        opts.write_buffer_size = 40000000
        opts.max_write_buffer_number = 50
        opts.max_background_compactions = 80
        opts.level0_file_num_compaction_trigger = 15
        opts.level0_slowdown_writes_trigger = 100
        opts.level0_stop_writes_trigger = 1000
        opts.target_file_size_base = 640000000
        opts.max_bytes_for_level_base = 640000000
        opts.min_write_buffer_number_to_merge = 30
        opts.stats_dump_period_sec = 360
        opts.table_cache_numshardbits =6

        opts.table_factory = rocksdb.BlockBasedTableFactory(
            filter_policy=rocksdb.BloomFilterPolicy(10),
            block_size=16384,
            block_cache=rocksdb.LRUCache(2 * (1024 ** 5)),
            block_cache_compressed=rocksdb.LRUCache(500 * (1024 ** 5)))
        self.DB = rocksdb.DB("/opt/ssd/addressdb", opts)
        self.Batch = rocksdb.WriteBatch()

The execution of the above code will be slower and slower, I want to call rocksdb to perform manual compression, but how to write this code?