twmht / python-rocksdb

Python bindings for RocksDB
BSD 3-Clause "New" or "Revised" License
274 stars 89 forks source link

How does python rocksdb execute the CreateDBStatistics method to collect information #86

Closed Jamiejoin closed 3 years ago

Jamiejoin commented 3 years ago

I now use python rocksdb to perform batch write operations. My machines are all SSD hard disks with 256G memory, but the writing speed will become slower and slower. I want to find specific problems. How to use this python package to get the status of rocksdb? Below is my code, I don’t know where to add the CreateDBStatistics method

    def connect_rocksdb(self):
        import rocksdb
        opts = rocksdb.Options()
        opts.create_if_missing = True
        opts.max_background_flushes = 15
        opts.max_open_files = 300000
        opts.write_buffer_size = 4000000
        opts.max_write_buffer_number = 15
        opts.max_background_compactions = 15
        opts.level0_file_num_compaction_trigger = 4
        opts.level0_slowdown_writes_trigger = 15
        opts.level0_stop_writes_trigger = 20
        opts.target_file_size_base = 640000000
        opts.max_bytes_for_level_base = 6400000000
        opts.table_factory = rocksdb.BlockBasedTableFactory(
            filter_policy=rocksdb.BloomFilterPolicy(10),
            block_size=16384,
            block_cache=rocksdb.LRUCache(2 * (1024 ** 5)),
            block_cache_compressed=rocksdb.LRUCache(500 * (1024 ** 5)))
       self.DB = rocksdb.DB("/opt/ssd/addressdb", opts)
       self.Batch = rocksdb.WriteBatch()

I even put all rocksdb directly on the tmpfs memory disk without using physical disks at all. I added statistical time to the code, but the batch write of rocksdb will become slower and slower if it is still executed.

                 starttime = datetime.datetime.now()
                 self.DB.write(self.Batch)
                 endtime = datetime.datetime.now()
                 print(endtime-starttime)