twmht / python-rocksdb

Python bindings for RocksDB
BSD 3-Clause "New" or "Revised" License
276 stars 88 forks source link

Regarding the issue of writing stalls #88

Closed Jamiejoin closed 3 years ago

Jamiejoin commented 3 years ago

Regarding the issue of stalling, I hope to get some help, I have spent 3 weeks on this issue and still have not been resolved. I understand that the issue should not be posted here, but I have not been resolved when I ask for help elsewhere,Please allow me to spend a few minutes to introduce my rocksdb problem

I used to put rocksdb on the ssd disk, but there will be a problem of slower and slower writing. In order to solve the problem, I expanded the memory of the server to 256GB, and put all the rocksdb data in the memory, using the tmpfs memory disk of linux , But the problem still exists, and the speed will be slower and slower,I suspect it is caused by the use of batch writing, because the separate put method I use will not have this problem,Below is my parameter configuration

2020/11/28-02:17:08.891198 7f5b14ed9d48 RocksDB version: 5.18.4
2020/11/28-02:17:08.891235 7f5b14ed9d48 Git sha rocksdb_build_git_sha:3d5054f358faad1a4e76497547b023defd79cc6c
2020/11/28-02:17:08.891238 7f5b14ed9d48 Compile date Oct  1 2020
2020/11/28-02:17:08.891241 7f5b14ed9d48 DB SUMMARY
2020/11/28-02:17:08.891321 7f5b14ed9d48 CURRENT file:  CURRENT
2020/11/28-02:17:08.891324 7f5b14ed9d48 IDENTITY file:  IDENTITY
2020/11/28-02:17:08.891330 7f5b14ed9d48 MANIFEST file:  MANIFEST-251092 size: 1542439 Bytes
2020/11/28-02:17:08.891333 7f5b14ed9d48 SST files in /data/addressdb/ dir, Total Num: 28, files: 245408.sst 24
5409.sst 245410.sst 245411.sst 245412.sst 245413.sst 245414.sst 245415.sst 245416.sst 
2020/11/28-02:17:08.891376 7f5b14ed9d48 Write Ahead Log file in /data/addressdb: 266325.log size: 268740975 ; 
266321.log size: 268719865 ; 
2020/11/28-02:17:08.891383 7f5b14ed9d48                         Options.error_if_exists: 0
2020/11/28-02:17:08.891386 7f5b14ed9d48                       Options.create_if_missing: 1
2020/11/28-02:17:08.891389 7f5b14ed9d48                         Options.paranoid_checks: 1
2020/11/28-02:17:08.891392 7f5b14ed9d48                                     Options.env: 0x7f5b13c009e0
2020/11/28-02:17:08.891395 7f5b14ed9d48                                Options.info_log: 0x55bd13a1f140
2020/11/28-02:17:08.891397 7f5b14ed9d48                Options.max_file_opening_threads: 16
2020/11/28-02:17:08.891400 7f5b14ed9d48                              Options.statistics: 0
2020/11/28-02:17:08.891402 7f5b14ed9d48                               Options.use_fsync: 1
2020/11/28-02:17:08.891404 7f5b14ed9d48                       Options.max_log_file_size: 0
2020/11/28-02:17:08.891407 7f5b14ed9d48                  Options.max_manifest_file_size: 1073741824
2020/11/28-02:17:08.891409 7f5b14ed9d48                   Options.log_file_time_to_roll: 0
2020/11/28-02:17:08.891412 7f5b14ed9d48                       Options.keep_log_file_num: 1000
2020/11/28-02:17:08.891414 7f5b14ed9d48                    Options.recycle_log_file_num: 0
2020/11/28-02:17:08.891417 7f5b14ed9d48                         Options.allow_fallocate: 1
2020/11/28-02:17:08.891420 7f5b14ed9d48                        Options.allow_mmap_reads: 0
2020/11/28-02:17:08.891422 7f5b14ed9d48                       Options.allow_mmap_writes: 0
2020/11/28-02:17:08.891424 7f5b14ed9d48                        Options.use_direct_reads: 0
2020/11/28-02:17:08.891427 7f5b14ed9d48                        Options.use_direct_io_for_flush_and_compaction: 0
2020/11/28-02:17:08.891429 7f5b14ed9d48          Options.create_missing_column_families: 0
2020/11/28-02:17:08.891432 7f5b14ed9d48                              Options.db_log_dir: 
2020/11/28-02:17:08.891434 7f5b14ed9d48                                 Options.wal_dir: /data/addressdb
2020/11/28-02:17:08.891436 7f5b14ed9d48                Options.table_cache_numshardbits: 6
2020/11/28-02:17:08.891439 7f5b14ed9d48                      Options.max_subcompactions: 1
2020/11/28-02:17:08.891441 7f5b14ed9d48                  Options.max_background_flushes: -1
2020/11/28-02:17:08.891443 7f5b14ed9d48                         Options.WAL_ttl_seconds: 0
2020/11/28-02:17:08.891447 7f5b14ed9d48                       Options.WAL_size_limit_MB: 0
2020/11/28-02:17:08.891449 7f5b14ed9d48             Options.manifest_preallocation_size: 4194304
2020/11/28-02:17:08.891451 7f5b14ed9d48                     Options.is_fd_close_on_exec: 1
2020/11/28-02:17:08.891454 7f5b14ed9d48                   Options.advise_random_on_open: 1
2020/11/28-02:17:08.891456 7f5b14ed9d48                    Options.db_write_buffer_size: 0
2020/11/28-02:17:08.891458 7f5b14ed9d48                    Options.write_buffer_manager: 0x55bd13a1ec00
2020/11/28-02:17:08.891461 7f5b14ed9d48         Options.access_hint_on_compaction_start: 1
2020/11/28-02:17:08.891463 7f5b14ed9d48  Options.new_table_reader_for_compaction_inputs: 0
2020/11/28-02:17:08.891465 7f5b14ed9d48           Options.random_access_max_buffer_size: 1048576
2020/11/28-02:17:08.891468 7f5b14ed9d48                      Options.use_adaptive_mutex: 0
2020/11/28-02:17:08.891470 7f5b14ed9d48                            Options.rate_limiter: 0
2020/11/28-02:17:08.891475 7f5b14ed9d48     Options.sst_file_manager.rate_bytes_per_sec: 0
2020/11/28-02:17:08.891477 7f5b14ed9d48                       Options.wal_recovery_mode: 2
2020/11/28-02:17:08.891480 7f5b14ed9d48                  Options.enable_thread_tracking: 0
2020/11/28-02:17:08.891482 7f5b14ed9d48                  Options.enable_pipelined_write: 0
2020/11/28-02:17:08.891484 7f5b14ed9d48         Options.allow_concurrent_memtable_write: 1
2020/11/28-02:17:08.891486 7f5b14ed9d48      Options.enable_write_thread_adaptive_yield: 1
2020/11/28-02:17:08.891489 7f5b14ed9d48             Options.write_thread_max_yield_usec: 100
2020/11/28-02:17:08.891491 7f5b14ed9d48            Options.write_thread_slow_yield_usec: 3
2020/11/28-02:17:08.891493 7f5b14ed9d48                               Options.row_cache: None
2020/11/28-02:17:08.891495 7f5b14ed9d48                              Options.wal_filter: None
2020/11/28-02:17:08.891498 7f5b14ed9d48             Options.avoid_flush_during_recovery: 0
2020/11/28-02:17:08.891500 7f5b14ed9d48             Options.allow_ingest_behind: 0
2020/11/28-02:17:08.891503 7f5b14ed9d48             Options.preserve_deletes: 0
2020/11/28-02:17:08.891506 7f5b14ed9d48             Options.two_write_queues: 0
2020/11/28-02:17:08.891508 7f5b14ed9d48             Options.manual_wal_flush: 0
2020/11/28-02:17:08.891510 7f5b14ed9d48             Options.max_background_jobs: 2
2020/11/28-02:17:08.891513 7f5b14ed9d48             Options.max_background_compactions: -1
2020/11/28-02:17:08.891515 7f5b14ed9d48             Options.avoid_flush_during_shutdown: 0
2020/11/28-02:17:08.891517 7f5b14ed9d48           Options.writable_file_max_buffer_size: 1048576
2020/11/28-02:17:08.891520 7f5b14ed9d48             Options.delayed_write_rate : 16777216
2020/11/28-02:17:08.891522 7f5b14ed9d48             Options.max_total_wal_size: 0
2020/11/28-02:17:08.891535 7f5b14ed9d48             Options.delete_obsolete_files_period_micros: 21600000000
2020/11/28-02:17:08.891538 7f5b14ed9d48                   Options.stats_dump_period_sec: 600
2020/11/28-02:17:08.891540 7f5b14ed9d48                          Options.max_open_files: -1
2020/11/28-02:17:08.891543 7f5b14ed9d48                          Options.bytes_per_sync: 0
2020/11/28-02:17:08.891546 7f5b14ed9d48                      Options.wal_bytes_per_sync: 0
2020/11/28-02:17:08.891548 7f5b14ed9d48       Options.compaction_readahead_size: 0
2020/11/28-02:17:08.891550 7f5b14ed9d48 Compression algorithms supported:
2020/11/28-02:17:08.891552 7f5b14ed9d48         kZSTDNotFinalCompression supported: 1
2020/11/28-02:17:08.891555 7f5b14ed9d48         kZSTD supported: 1
2020/11/28-02:17:08.891558 7f5b14ed9d48         kXpressCompression supported: 0
2020/11/28-02:17:08.891560 7f5b14ed9d48         kLZ4HCCompression supported: 1
2020/11/28-02:17:08.891562 7f5b14ed9d48         kLZ4Compression supported: 1
2020/11/28-02:17:08.891565 7f5b14ed9d48         kBZip2Compression supported: 1
2020/11/28-02:17:08.891567 7f5b14ed9d48         kZlibCompression supported: 1
2020/11/28-02:17:08.891569 7f5b14ed9d48         kSnappyCompression supported: 1
2020/11/28-02:17:08.891577 7f5b14ed9d48 Fast CRC32 supported: Supported on x86
2020/11/28-02:17:08.891642 7f5b14ed9d48 [/version_set.cc:3507] Recovering from manifest file: MANIFEST-251092
2020/11/28-02:17:08.891974 7f5b14ed9d48 [/column_family.cc:472] --------------- Options for column family [defa
ult]:
2020/11/28-02:17:08.891977 7f5b14ed9d48               Options.comparator: leveldb.BytewiseComparator
2020/11/28-02:17:08.891980 7f5b14ed9d48           Options.merge_operator: None
2020/11/28-02:17:08.891982 7f5b14ed9d48        Options.compaction_filter: None
2020/11/28-02:17:08.891984 7f5b14ed9d48        Options.compaction_filter_factory: None
2020/11/28-02:17:08.891987 7f5b14ed9d48         Options.memtable_factory: SkipListFactory
2020/11/28-02:17:08.891989 7f5b14ed9d48            Options.table_factory: BlockBasedTable
2020/11/28-02:17:08.892024 7f5b14ed9d48            table_factory options:   flush_block_policy_factory:  FlushBlockBySizePolicyFactory (0x55bd137dab80)
  cache_index_and_filter_blocks: 0
  cache_index_and_filter_blocks_with_high_priority: 0
  pin_l0_filter_and_index_blocks_in_cache: 0
  pin_top_level_index_and_filter: 1
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x55bd13a1bd30
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 8388608
    num_shard_bits : 4
    strict_capacity_limit : 0
    memory_allocator : None
    high_pri_pool_ratio: 0.000
  block_cache_compressed: 0
  persistent_cache: 0
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: nullptr
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0
2020/11/28-02:17:08.892029 7f5b14ed9d48        Options.write_buffer_size: 67108864
2020/11/28-02:17:08.892032 7f5b14ed9d48  Options.max_write_buffer_number: 2
2020/11/28-02:17:08.892034 7f5b14ed9d48          Options.compression: Snappy
2020/11/28-02:17:08.892036 7f5b14ed9d48                  Options.bottommost_compression: Disabled
2020/11/28-02:17:08.892039 7f5b14ed9d48       Options.prefix_extractor: nullptr
2020/11/28-02:17:08.892041 7f5b14ed9d48   Options.memtable_insert_with_hint_prefix_extractor: nullptr
2020/11/28-02:17:08.892043 7f5b14ed9d48             Options.num_levels: 7
2020/11/28-02:17:08.892046 7f5b14ed9d48        Options.min_write_buffer_number_to_merge: 1
2020/11/28-02:17:08.892048 7f5b14ed9d48     Options.max_write_buffer_number_to_maintain: 0
2020/11/28-02:17:08.892050 7f5b14ed9d48            Options.bottommost_compression_opts.window_bits: -14
2020/11/28-02:17:08.892053 7f5b14ed9d48                  Options.bottommost_compression_opts.level: 32767
2020/11/28-02:17:08.892055 7f5b14ed9d48               Options.bottommost_compression_opts.strategy: 0
2020/11/28-02:17:08.892069 7f5b14ed9d48         Options.bottommost_compression_opts.max_dict_bytes: 0
2020/11/28-02:17:08.892071 7f5b14ed9d48         Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2020/11/28-02:17:08.892073 7f5b14ed9d48                  Options.bottommost_compression_opts.enabled: false
2020/11/28-02:17:08.892075 7f5b14ed9d48            Options.compression_opts.window_bits: -14
2020/11/28-02:17:08.892078 7f5b14ed9d48                  Options.compression_opts.level: 32767
2020/11/28-02:17:08.892080 7f5b14ed9d48               Options.compression_opts.strategy: 0
2020/11/28-02:17:08.892082 7f5b14ed9d48         Options.compression_opts.max_dict_bytes: 0
2020/11/28-02:17:08.892084 7f5b14ed9d48         Options.compression_opts.zstd_max_train_bytes: 0
2020/11/28-02:17:08.892086 7f5b14ed9d48                  Options.compression_opts.enabled: false
2020/11/28-02:17:08.892088 7f5b14ed9d48      Options.level0_file_num_compaction_trigger: 4
2020/11/28-02:17:08.892090 7f5b14ed9d48          Options.level0_slowdown_writes_trigger: 20
2020/11/28-02:17:08.892093 7f5b14ed9d48              Options.level0_stop_writes_trigger: 36
2020/11/28-02:17:08.892095 7f5b14ed9d48                   Options.target_file_size_base: 33554432
2020/11/28-02:17:08.892097 7f5b14ed9d48             Options.target_file_size_multiplier: 1
2020/11/28-02:17:08.892100 7f5b14ed9d48                Options.max_bytes_for_level_base: 268435456
2020/11/28-02:17:08.892102 7f5b14ed9d48 Options.level_compaction_dynamic_level_bytes: 0
2020/11/28-02:17:08.892104 7f5b14ed9d48          Options.max_bytes_for_level_multiplier: 10.000000
2020/11/28-02:17:08.892106 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[0]: 1
2020/11/28-02:17:08.892109 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[1]: 1
2020/11/28-02:17:08.892111 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[2]: 1
2020/11/28-02:17:08.892113 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[3]: 1
2020/11/28-02:17:08.892116 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[4]: 1
2020/11/28-02:17:08.892118 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[5]: 1
2020/11/28-02:17:08.892122 7f5b14ed9d48 Options.max_bytes_for_level_multiplier_addtl[6]: 1
2020/11/28-02:17:08.892124 7f5b14ed9d48       Options.max_sequential_skip_in_iterations: 8
2020/11/28-02:17:08.892126 7f5b14ed9d48                    Options.max_compaction_bytes: 838860800
2020/11/28-02:17:08.892128 7f5b14ed9d48                        Options.arena_block_size: 8388608
2020/11/28-02:17:08.892131 7f5b14ed9d48   Options.soft_pending_compaction_bytes_limit: 68719476736
2020/11/28-02:17:08.892133 7f5b14ed9d48   Options.hard_pending_compaction_bytes_limit: 274877906944
2020/11/28-02:17:08.892135 7f5b14ed9d48       Options.rate_limit_delay_max_milliseconds: 100
2020/11/28-02:17:08.892137 7f5b14ed9d48                Options.disable_auto_compactions: 0
2020/11/28-02:17:08.892140 7f5b14ed9d48                        Options.compaction_style: kCompactionStyleLevel
2020/11/28-02:17:08.892142 7f5b14ed9d48                          Options.compaction_pri: kByCompensatedSize
2020/11/28-02:17:08.892144 7f5b14ed9d48 Options.compaction_options_universal.size_ratio: 1
2020/11/28-02:17:08.892147 7f5b14ed9d48 Options.compaction_options_universal.min_merge_width: 2
2020/11/28-02:17:08.892149 7f5b14ed9d48 Options.compaction_options_universal.max_merge_width: 4294967295
2020/11/28-02:17:08.892152 7f5b14ed9d48 Options.compaction_options_universal.max_size_amplification_percent: 200
2020/11/28-02:17:08.892154 7f5b14ed9d48 Options.compaction_options_universal.compression_size_percent: -1
2020/11/28-02:17:08.892156 7f5b14ed9d48 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2020/11/28-02:17:08.892158 7f5b14ed9d48 Options.compaction_options_fifo.max_table_files_size: 1073741824
2020/11/28-02:17:08.892161 7f5b14ed9d48 Options.compaction_options_fifo.allow_compaction: 0
2020/11/28-02:17:08.892163 7f5b14ed9d48 Options.compaction_options_fifo.ttl: 0
2020/11/28-02:17:08.892165 7f5b14ed9d48                   Options.table_properties_collectors: 
2020/11/28-02:17:08.892167 7f5b14ed9d48                   Options.inplace_update_support: 0
2020/11/28-02:17:08.892169 7f5b14ed9d48                 Options.inplace_update_num_locks: 10000
2020/11/28-02:17:08.892172 7f5b14ed9d48               Options.memtable_prefix_bloom_size_ratio: 0.000000
2020/11/28-02:17:08.892174 7f5b14ed9d48   Options.memtable_huge_page_size: 0
2020/11/28-02:17:08.892177 7f5b14ed9d48                           Options.bloom_locality: 0
2020/11/28-02:17:08.892179 7f5b14ed9d48                    Options.max_successive_merges: 0
2020/11/28-02:17:08.892181 7f5b14ed9d48                Options.optimize_filters_for_hits: 0
2020/11/28-02:17:08.892183 7f5b14ed9d48                Options.paranoid_file_checks: 0
2020/11/28-02:17:08.892185 7f5b14ed9d48                Options.force_consistency_checks: 0
2020/11/28-02:17:08.892187 7f5b14ed9d48                Options.report_bg_io_stats: 0
2020/11/28-02:17:08.892189 7f5b14ed9d48                               Options.ttl: 0

In order to ensure the consistency of writing, I used batch writing and printing time, but this time will get longer and longer. It only needs 0:00:00.00728 to start the program at the beginning, as the program runs time It will slowly become 00:00:10 or even longer,Cause me to restart the program regularly,I don’t understand what I did wrong. I have replaced all of them with memory, but the efficiency cannot be improved. Please help me, thank you very much

                starttime = datetime.datetime.now()
                self.DB.write(rocksdb.WriteBatch())
                endtime = datetime.datetime.now()
                print(endtime - starttime)

This is the state that costs 00:00:07.128520

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      2/0   37.57 MB   0.5      0.0     0.0      0.0       5.0      5.0       0.0   1.0      0.0      6.1       843       364    2.316       0      0
  L1      3/0   82.26 MB   0.3     12.0     5.0      7.0       7.0      0.0       0.0   1.4     92.3     53.9       133        91    1.462    300M   124M
  L2      2/0   55.75 MB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
  L5      3/0    1.72 MB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
  L6     22/0   135.73 MB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
 Sum     32/0   313.03 MB   0.0     12.0     5.0      7.0      12.0      5.0       0.0   2.4     12.6     12.6       976       455    2.146    300M   124M
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
Uptime(secs): 15217.4 total, 0.0 interval
Flush(GB): cumulative 5.033, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 12.04 GB write, 0.81 MB/s write, 11.99 GB read, 0.81 MB/s read, 976.2 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_com
iFA88 commented 3 years ago

Hey! Do you use some cloud server? Maybe you have some recursion in your program code ?

I'm using this library for years, but this issue never happened to me.

Jamiejoin commented 3 years ago

Hey! Do you use some cloud server? Maybe you have some recursion in your program code ?

I'm using this library for years, but this issue never happened to me.

I don’t use cloud services, and the server hardware configuration is very high. I just used the rocksdb.WriteBatch() method. I suspect this is the problem.

Jamiejoin commented 3 years ago

Hey! Do you use some cloud server? Maybe you have some recursion in your program code ?

I'm using this library for years, but this issue never happened to me.

Do you use rocksdb.WriteBatch() this method?

iFA88 commented 3 years ago

Please share for me a most small code what reproduces this issue. Yes I use WriteBatch a lot. My dbs are 500-700gb bigs.

Jamiejoin commented 3 years ago

Please share for me a most small code what reproduces this issue. Yes I use WriteBatch a lot. My dbs are 500-700gb bigs.

The following code can make the writing time longer and longer

# coding:utf-8
import rocksdb
import string
import random
import datetime

class TestR():
    def __init__(self):
        pass

    def connect_rocksdb(self):
        opts = rocksdb.Options()
        opts.create_if_missing = True
        opts.max_background_flushes = 32
        opts.max_open_files = -1
        opts.write_buffer_size = 671088640
        opts.max_write_buffer_number = 16
        opts.max_background_compactions = 32
        opts.level0_file_num_compaction_trigger = 4
        opts.level0_slowdown_writes_trigger = 20
        opts.level0_stop_writes_trigger = 36
        opts.target_file_size_base = 33554432
        opts.target_file_size_multiplier = 1
        opts.max_bytes_for_level_base = 268435456
        opts.max_bytes_for_level_multiplier = 10
        opts.min_write_buffer_number_to_merge = 10
        opts.arena_block_size = 8388608
        opts.use_fsync = True
        opts.stats_dump_period_sec = 100
        opts.allow_mmap_reads = True
        opts.compression = rocksdb.CompressionType.snappy_compression
        opts.table_factory = rocksdb.BlockBasedTableFactory(
             filter_policy=rocksdb.BloomFilterPolicy(10),
             block_size=166384,
             block_cache=rocksdb.LRUCache(2 * (1024 ** 2)),
             block_cache_compressed=rocksdb.LRUCache(100 * (1024 ** 2)))
        self.DB = rocksdb.DB("/data/testdb/", opts)
        self.Batch = rocksdb.WriteBatch()

    def testwrite(self):
        self.connect_rocksdb()
        while True:
            for i in range(1,100):
                key = (''.join(random.sample(string.ascii_letters + string.digits, 32)))
                value = (''.join(random.sample(string.ascii_letters + string.digits, 32)))
                self.Batch.put(str(key).encode(),str(value).encode())
            starttime = datetime.datetime.now()
            self.DB.write(self.Batch)
            endtime = datetime.datetime.now()
            print(endtime - starttime)

t=TestR()
t.testwrite()
iFA88 commented 3 years ago

Yeah, after use of WriteBatch you should create a new one! In this situation like you have a buffer which is every time bigger and you write every time to the disk

Jamiejoin commented 3 years ago

Yeah, after use of WriteBatch you should create a new one! In this situation like you have a buffer which is every time bigger and you write every time to the disk

Do you mean I need to do del self.DB?

iFA88 commented 3 years ago

No! Just this:

self.DB.write(self.Batch)
self.Batch = rocksdb.WriteBatch()
Jamiejoin commented 3 years ago

No! Just this:

self.DB.write(self.Batch)
self.Batch = rocksdb.WriteBatch()

thank you