search

facebook / rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.
http://rocksdb.org
GNU General Public License v2.0
27.93k stars 6.22k forks source link

Solution to the periodic slowdown of GetUpdatesSnce #12516

Open 1522140840 opened 3 months ago

1522140840 commented 3 months ago

In my project, I use GetUpdatesSince for master-slave synchronization. I have noticed that the execution time of this method periodically increases, and by examining the source code, it is easy to identify that it is caused by SeekToStartSequence. Essentially, this is due to the periodic enlargement of the WAL (Write-Ahead Log) file. To address this issue, I have made a small optimization in my project. My approach involves adding a new method called GoOnRead to the TransactionLogIterator. As long as the WAL file hasn't switched and there is new data being written, I reset the state of the iterator to be valid. This way, I don't have to frequently reopen the iterator and can read the new data.

The overall time-consuming issue of my master-slave synchronization with TP999 is as follows: [The program itself sleeps for 100ms.]

image
MalikHou commented 1 month ago

Perhaps you can cache the iterator directly and create it when it returns not found. this is the recommended method.

1522140840 commented 1 month ago

Perhaps you can cache the iterator directly and create it when it returns not found. this is the recommended method.

When the iterator data is invalidated after being read, newly written data cannot be read; cached iterators cannot solve the problem of new data.