How to achieve ~20% replication throughput improvement using Read Free Replication (RFR) feature

[hanxu12]

Background

• According to the RFR pull request, RFR improves update replication throughput by ~20%.
• The machine I use are 3 C220g1 nodes: Two E5-2630 CPU 128 GB Memory 1.2 TB 10K RPM Hard disk 10GB Intel NIC *Ubuntu 20LTS

Setup

• I try to reproduce it on my end, but the improvement was ~1-5% by comparing the Percona 8.0 setup below.
• I have 1 leader and 2 replicas (all using MyRocks engine, 1 replica with rocksdb_read_free_rpl variable set to PK_SK, and the other replica as OFF). The leader runs with sysbench oltp_update_non_index and oltp_update_index scripts. For example, sysbench /usr/share/sysbench/oltp_update_non_index.lua --mysql-user=root --mysql-password=123456 --mysql-db=test3 --threads=1000 --time=20 run sysbench /usr/share/sysbench/oltp_update_index.lua --mysql-user=root --mysql-password=123456 --mysql-db=test2 --threads=64 --events=2500 --time=0 run
• The table is 20M, 1 thread on each replica, set rocksdb_block_cache_size=64G and performed lookup to ensure all rows are in memory.

Result

1. It takes each replica around 220s to catch up with ~10000 updates on the server. So the replication throughput is ~45 updates/s, which is far less as compared to 5600-6800 row_updated/s.
2. By generating different workload, usually I can only achieve ~1-3% throughput improvement. Is there anything wrong with my setup?