iFA88
commented
3 years ago Hey! Do you use some cloud server? Maybe you have some recursion in your program code ?
I'm using this library for years, but this issue never happened to me.
Closed Jamiejoin closed 3 years ago
iFA88
commented
3 years ago Hey! Do you use some cloud server? Maybe you have some recursion in your program code ?
I'm using this library for years, but this issue never happened to me.
Jamiejoin
commented
3 years ago Hey! Do you use some cloud server? Maybe you have some recursion in your program code ?
I'm using this library for years, but this issue never happened to me.
I don’t use cloud services, and the server hardware configuration is very high. I just used the rocksdb.WriteBatch() method. I suspect this is the problem.
Jamiejoin
commented
3 years ago Hey! Do you use some cloud server? Maybe you have some recursion in your program code ?
I'm using this library for years, but this issue never happened to me.
Do you use rocksdb.WriteBatch() this method？
iFA88
commented
3 years ago Please share for me a most small code what reproduces this issue.
Yes I use WriteBatch a lot. My dbs are 500-700gb bigs.
Jamiejoin
commented
3 years ago Please share for me a most small code what reproduces this issue. Yes I use
WriteBatcha lot. My dbs are 500-700gb bigs.
The following code can make the writing time longer and longer
# coding:utf-8
import rocksdb
import string
import random
import datetime
class TestR():
def __init__(self):
pass
def connect_rocksdb(self):
opts = rocksdb.Options()
opts.create_if_missing = True
opts.max_background_flushes = 32
opts.max_open_files = -1
opts.write_buffer_size = 671088640
opts.max_write_buffer_number = 16
opts.max_background_compactions = 32
opts.level0_file_num_compaction_trigger = 4
opts.level0_slowdown_writes_trigger = 20
opts.level0_stop_writes_trigger = 36
opts.target_file_size_base = 33554432
opts.target_file_size_multiplier = 1
opts.max_bytes_for_level_base = 268435456
opts.max_bytes_for_level_multiplier = 10
opts.min_write_buffer_number_to_merge = 10
opts.arena_block_size = 8388608
opts.use_fsync = True
opts.stats_dump_period_sec = 100
opts.allow_mmap_reads = True
opts.compression = rocksdb.CompressionType.snappy_compression
opts.table_factory = rocksdb.BlockBasedTableFactory(
filter_policy=rocksdb.BloomFilterPolicy(10),
block_size=166384,
block_cache=rocksdb.LRUCache(2 * (1024 ** 2)),
block_cache_compressed=rocksdb.LRUCache(100 * (1024 ** 2)))
self.DB = rocksdb.DB("/data/testdb/", opts)
self.Batch = rocksdb.WriteBatch()
def testwrite(self):
self.connect_rocksdb()
while True:
for i in range(1,100):
key = (''.join(random.sample(string.ascii_letters + string.digits, 32)))
value = (''.join(random.sample(string.ascii_letters + string.digits, 32)))
self.Batch.put(str(key).encode(),str(value).encode())
starttime = datetime.datetime.now()
self.DB.write(self.Batch)
endtime = datetime.datetime.now()
print(endtime - starttime)
t=TestR()
t.testwrite()Yeah, after use of WriteBatch you should create a new one!
In this situation like you have a buffer which is every time bigger and you write every time to the disk
Jamiejoin
commented
3 years ago Yeah, after use of
WriteBatchyou should create a new one! In this situation like you have a buffer which is every time bigger and you write every time to the disk
Do you mean I need to do del self.DB?
iFA88
commented
3 years ago No! Just this:
self.DB.write(self.Batch)
self.Batch = rocksdb.WriteBatch()No! Just this:
self.DB.write(self.Batch) self.Batch = rocksdb.WriteBatch()
thank you
Regarding the issue of stalling, I hope to get some help, I have spent 3 weeks on this issue and still have not been resolved. I understand that the issue should not be posted here, but I have not been resolved when I ask for help elsewhere，Please allow me to spend a few minutes to introduce my rocksdb problem
I used to put rocksdb on the ssd disk, but there will be a problem of slower and slower writing. In order to solve the problem, I expanded the memory of the server to 256GB, and put all the rocksdb data in the memory, using the tmpfs memory disk of linux , But the problem still exists, and the speed will be slower and slower，I suspect it is caused by the use of batch writing, because the separate put method I use will not have this problem，Below is my parameter configuration
In order to ensure the consistency of writing, I used batch writing and printing time, but this time will get longer and longer. It only needs 0:00:00.00728 to start the program at the beginning, as the program runs time It will slowly become 00:00:10 or even longer，Cause me to restart the program regularly，I don’t understand what I did wrong. I have replaced all of them with memory, but the efficiency cannot be improved. Please help me, thank you very much
This is the state that costs 00:00:07.128520