apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store
https://apple.github.io/foundationdb/
Apache License 2.0
14.18k stars 1.29k forks source link

Remove two ptree searches when processing a clear #11435

Closed atn34 closed 1 month ago

atn34 commented 1 month ago

Also update a previous use of the "getting lower_bound from lastLessOrEqual" technique to use the same pattern.

Performance testing (using mako with a point clear workload, or something similar) pending local compilation finishing. Mostly relying on the CI infrastructure for correctness testing.

Setup: mac m1 2020, single-process fdbserver using ssd storage engine. cpu usage hovered around 100%.

Summary: sets went from 17165 TPS to 17455 TPS, and clears went from 18292 TPS to 18941 TPS.

Before

$ DYLD_LIBRARY_PATH=lib/ bin/mako --mode=run -xo1 --rows 1000000 -s 60 --async_xacts 400 > set.txt & DYLD_LIBRARY_PATH=lib/ bin/mako --mode=run -xc1 --rows 1000000 -s 60 --async_xacts 400 > clear.txt ;wait; cat set.txt clear.txt
[1] 85010
[1]  + done       DYLD_LIBRARY_PATH=lib/ bin/mako --mode=run -xo1 --rows 1000000 -s 60  400 > 
                OVERWRITE          TPS  Conflicts/s 
============ ============ ============ ============ 
         OPS        27498     26661.98         0.00 
         OPS        27855     26129.68         0.00 
         OPS        27887     26862.64         0.00 
         OPS        28507     27834.22         0.00 
         OPS        27749     27106.23         0.00 
         OPS        28096     27529.10         0.00 
         OPS        23044     22443.47         0.00 
         OPS         8558      8377.08         0.00 
         OPS         7599      7377.63         0.00 
         OPS         7740      7562.86         0.00 
         OPS         7374      7205.08         0.00 
         OPS         7325      7162.60         0.00 
         OPS         7140      6981.52         0.00 
         OPS        23689     23064.04         0.00 
         OPS        23231     22497.91         0.00 
         OPS        23278     22651.26         0.00 
         OPS        23620     23012.19         0.00 
         OPS        22478     21939.16         0.00 
         OPS        13358     12993.47         0.00 
         OPS        12436     12155.86         0.00 
         OPS        14767     14437.25         0.00 
         OPS         6928      6723.91         0.00 
         OPS        14839     14410.77         0.00 
         OPS        19204     18775.01         0.00 
         OPS        18324     17916.57         0.00 
         OPS        17356     16947.38         0.00 
         OPS        21330     20817.61         0.00 
         OPS        18785     18281.72         0.00 
         OPS        18255     17800.46         0.00 
         OPS        14065     13688.46         0.00 
         OPS        16760     16385.34         0.00 
         OPS        19038     18545.48         0.00 
         OPS        15266     14925.76         0.00 
         OPS        15012     14690.85         0.00 
         OPS        17564     17149.53         0.00 
         OPS        19949     19497.08         0.00 
         OPS        14489     14168.25         0.00 
         OPS        17559     17179.75         0.00 
         OPS        20234     19774.84         0.00 
         OPS        15153     14785.75         0.00 
         OPS        20521     20050.85         0.00 
         OPS        14578     14235.53         0.00 
         OPS        16403     15979.37         0.00 
         OPS        15160     14821.07         0.00 
         OPS        20625     20123.69         0.00 
         OPS        16254     15870.17         0.00 
         OPS        15734     15368.18         0.00 
         OPS        17221     16820.93         0.00 
         OPS        19242     18753.24         0.00 
         OPS        16766     16323.25         0.00 
         OPS        15533     15197.28         0.00 
         OPS        17664     17235.57         0.00 
         OPS        18899     18450.51         0.00 
         OPS        15370     15050.84         0.00 
         OPS        15125     14785.66         0.00 
         OPS        17575     17100.73         0.00 
         OPS        18703     18266.87         0.00 
         OPS        16056     15647.23         0.00 

====== Total Duration 60.100 sec ======

Total Processes:          1
Total Threads:            1
Total Async Xacts:      400
Target TPS:               0
Total Xacts:        1031646
Total Conflicts:          0
Total Errors:             0
Total Timeouts:           0
Overall TPS:          17165

%CPU Worker Processes:          16.93 
%CPU Worker Threads:             3.12 
%CPU Local Network Threads:     13.32 
%CPU External Network Threads:   0.50 

                OVERWRITE       COMMIT          TPS  Conflicts/s 
============ ============ ============ ============ ============ 
Total OPS         1031646      1031646     17165.42         0.00 
Errors                  0            0 
Timeouts                0            0 

Latency (us)   OVERWRITE       COMMIT  TRANSACTION 
============ ============ ============ ============ 
Samples              1200         1200         1200 
Min                     0        15915        15918 
Avg                     0        34555        34826 
Max                   N/A        58747        58753 
Median                  0        28979        29856 
95.0 pctile             0        28979        29856 
99.0 pctile             0        28979        29856 
99.9 pctile             0        28979        29856 
                    CLEAR          TPS  Conflicts/s 
============ ============ ============ ============ 
         OPS        31598     31014.14         0.00 
         OPS        30953     29159.03         0.00 
         OPS        31740     30629.03         0.00 
         OPS        32281     31476.03         0.00 
         OPS        32260     31535.25         0.00 
         OPS        30905     30215.70         0.00 
         OPS        24907     24330.70         0.00 
         OPS         7974      7776.24         0.00 
         OPS         7364      7176.25         0.00 
         OPS         7537      7325.12         0.00 
         OPS         7205      7041.11         0.00 
         OPS         7287      7115.94         0.00 
         OPS         7852      7650.92         0.00 
         OPS        25716     25062.89         0.00 
         OPS        25706     25082.64         0.00 
         OPS        24581     23999.71         0.00 
         OPS        25678     25117.67         0.00 
         OPS        23635     23071.75         0.00 
         OPS        14634     14310.73         0.00 
         OPS        13253     12939.01         0.00 
         OPS        15207     14899.89         0.00 
         OPS         7433      7276.74         0.00 
         OPS        15012     14701.38         0.00 
         OPS        19295     18904.90         0.00 
         OPS        20390     19927.35         0.00 
         OPS        19344     18864.06         0.00 
         OPS        24062     23497.55         0.00 
         OPS        18267     17849.25         0.00 
         OPS        20708     20252.89         0.00 
         OPS        13268     12962.63         0.00 
         OPS        16254     15880.99         0.00 
         OPS        19169     18731.30         0.00 
         OPS        15493     15124.90         0.00 
         OPS        15760     15407.88         0.00 
         OPS        17642     17239.04         0.00 
         OPS        21691     21190.74         0.00 
         OPS        17958     17534.49         0.00 
         OPS        16263     15917.88         0.00 
         OPS        22433     21832.80         0.00 
         OPS        14082     13768.20         0.00 
         OPS        21568     21123.78         0.00 
         OPS        15097     14755.82         0.00 
         OPS        16202     15742.70         0.00 
         OPS        15117     14778.33         0.00 
         OPS        21363     20930.11         0.00 
         OPS        19207     18733.01         0.00 
         OPS        17389     16953.42         0.00 
         OPS        17669     17218.60         0.00 
         OPS        21063     20494.88         0.00 
         OPS        18402     18012.75         0.00 
         OPS        15547     15193.07         0.00 
         OPS        17103     16688.20         0.00 
         OPS        21141     20679.34         0.00 
         OPS        16939     16538.92         0.00 
         OPS        16610     16219.03         0.00 
         OPS        17123     16753.89         0.00 
         OPS        20667     20112.37         0.00 
         OPS        17081     16676.60         0.00 

====== Total Duration 60.145 sec ======

Total Processes:          1
Total Threads:            1
Total Async Xacts:      400
Target TPS:               0
Total Xacts:        1100179
Total Conflicts:          0
Total Errors:             0
Total Timeouts:           0
Overall TPS:          18292

%CPU Worker Processes:          18.47 
%CPU Worker Threads:             3.29 
%CPU Local Network Threads:     14.67 
%CPU External Network Threads:   0.50 

                    CLEAR       COMMIT          TPS  Conflicts/s 
============ ============ ============ ============ ============ 
Total OPS         1100179      1100179     18292.01         0.00 
Errors                  0            0 
Timeouts                0            0 

Latency (us)       CLEAR       COMMIT  TRANSACTION 
============ ============ ============ ============ 
Samples              1200         1200         1200 
Min                     0        24275        25036 
Avg                     0        45559        45819 
Max                   N/A        77513        77517 
Median                  0        34959        34959 
95.0 pctile             0        34959        34959 
99.0 pctile             0        34959        34959 
99.9 pctile             0        34959        34959

After

$ DYLD_LIBRARY_PATH=lib/ bin/mako --mode=run -xo1 --rows 1000000 -s 60 --async_xacts 400 > set.txt & DYLD_LIBRARY_PATH=lib/ bin/mako --mode=run -xc1 --rows 1000000 -s 60 --async_xacts 400 > clear.txt ;wait; cat set.txt clear.txt
[1] 84651
[1]  + done       DYLD_LIBRARY_PATH=lib/ bin/mako --mode=run -xo1 --rows 1000000 -s 60  400 > 
                OVERWRITE          TPS  Conflicts/s 
============ ============ ============ ============ 
         OPS        28825     28223.37         0.00 
         OPS        26847     25250.35         0.00 
         OPS        27897     27063.83         0.00 
         OPS        26720     25904.46         0.00 
         OPS        26891     26136.02         0.00 
         OPS        14304     13869.30         0.00 
         OPS        11153     10707.18         0.00 
         OPS        11075     10821.46         0.00 
         OPS        13528     13076.25         0.00 
         OPS         7977      7798.41         0.00 
         OPS        19164     18674.18         0.00 
         OPS        23852     23233.73         0.00 
         OPS        18825     18362.38         0.00 
         OPS        22206     21607.42         0.00 
         OPS        20660     20196.73         0.00 
         OPS        15798     15416.82         0.00 
         OPS        14348     14040.74         0.00 
         OPS        17638     17204.86         0.00 
         OPS        16103     15630.27         0.00 
         OPS        16756     16338.83         0.00 
         OPS        13798     13269.39         0.00 
         OPS        20384     19820.63         0.00 
         OPS        18750     18325.18         0.00 
         OPS        20689     20257.23         0.00 
         OPS        17691     17180.18         0.00 
         OPS        16316     15980.79         0.00 
         OPS        18283     17808.61         0.00 
         OPS        16246     15838.48         0.00 
         OPS        16888     16478.89         0.00 
         OPS        15652     15229.82         0.00 
         OPS        15960     15581.75         0.00 
         OPS        19546     19066.55         0.00 
         OPS        16668     16229.91         0.00 
         OPS        16913     16531.09         0.00 
         OPS        20151     19614.47         0.00 
         OPS        16928     16523.47         0.00 
         OPS        18965     18512.32         0.00 
         OPS        16145     15751.66         0.00 
         OPS        14322     13921.93         0.00 
         OPS        15151     14757.75         0.00 
         OPS        18571     18137.91         0.00 
         OPS        15278     14933.17         0.00 
         OPS        16060     15693.34         0.00 
         OPS        22243     21699.60         0.00 
         OPS        15349     14883.12         0.00 
         OPS        18125     17691.68         0.00 
         OPS        16654     16228.16         0.00 
         OPS        16933     16351.83         0.00 
         OPS        12867     12574.97         0.00 
         OPS        22321     21715.66         0.00 
         OPS        14416     14087.75         0.00 
         OPS        16638     16251.21         0.00 
         OPS        20506     20068.29         0.00 
         OPS        19141     18638.88         0.00 
         OPS        14897     14524.89         0.00 
         OPS        22250     21703.54         0.00 
         OPS        15704     15372.04         0.00 
         OPS        17417     16995.46         0.00 

====== Total Duration 60.072 sec ======

Total Processes:          1
Total Threads:            1
Total Async Xacts:      400
Target TPS:               0
Total Xacts:        1048599
Total Conflicts:          0
Total Errors:             0
Total Timeouts:           0
Overall TPS:          17455

%CPU Worker Processes:          17.03 
%CPU Worker Threads:             3.16 
%CPU Local Network Threads:     13.39 
%CPU External Network Threads:   0.48 

                OVERWRITE       COMMIT          TPS  Conflicts/s 
============ ============ ============ ============ ============ 
Total OPS         1048599      1048599     17455.65         0.00 
Errors                  0            0 
Timeouts                0            0 

Latency (us)   OVERWRITE       COMMIT  TRANSACTION 
============ ============ ============ ============ 
Samples              1200         1200         1200 
Min                     0        15301        15304 
Avg                     0        26430        26611 
Max                   N/A        45719        45797 
Median                  0        18184        18731 
95.0 pctile             0        18184        18731 
99.0 pctile             0        18184        18731 
99.9 pctile             0        18184        18731 
                    CLEAR          TPS  Conflicts/s 
============ ============ ============ ============ 
         OPS        32465     31928.53         0.00 
         OPS        29653     27842.56         0.00 
         OPS        30854     29970.18         0.00 
         OPS        32981     32040.86         0.00 
         OPS        31129     30227.17         0.00 
         OPS        15959     15469.98         0.00 
         OPS        11061     10781.30         0.00 
         OPS        12454     12145.97         0.00 
         OPS        14711     14324.53         0.00 
         OPS         8000      7790.07         0.00 
         OPS        20147     19726.56         0.00 
         OPS        26773     26039.75         0.00 
         OPS        20731     20196.62         0.00 
         OPS        23893     23343.74         0.00 
         OPS        22000     21509.80         0.00 
         OPS        17932     17441.95         0.00 
         OPS        15060     14743.04         0.00 
         OPS        17876     17451.11         0.00 
         OPS        16457     16065.45         0.00 
         OPS        18286     17860.25         0.00 
         OPS        14961     14615.39         0.00 
         OPS        21491     21030.41         0.00 
         OPS        20622     20078.99         0.00 
         OPS        21320     20787.64         0.00 
         OPS        20558     20102.84         0.00 
         OPS        16581     16155.17         0.00 
         OPS        18007     17619.65         0.00 
         OPS        16526     16126.97         0.00 
         OPS        19678     19226.62         0.00 
         OPS        16465     16084.28         0.00 
         OPS        16884     16441.17         0.00 
         OPS        21443     20910.07         0.00 
         OPS        19088     18602.21         0.00 
         OPS        17812     17426.29         0.00 
         OPS        21971     21449.42         0.00 
         OPS        18578     18141.27         0.00 
         OPS        21218     20760.74         0.00 
         OPS        18000     17513.63         0.00 
         OPS        14491     14113.76         0.00 
         OPS        16194     15797.96         0.00 
         OPS        21253     20750.47         0.00 
         OPS        18049     17573.55         0.00 
         OPS        15862     15511.16         0.00 
         OPS        24809     24296.73         0.00 
         OPS        14759     14408.80         0.00 
         OPS        20148     19665.32         0.00 
         OPS        18915     18434.67         0.00 
         OPS        15627     15297.55         0.00 
         OPS        14927     14532.68         0.00 
         OPS        24688     24152.62         0.00 
         OPS        15173     14805.27         0.00 
         OPS        16630     16212.19         0.00 
         OPS        20294     19882.50         0.00 
         OPS        22162     21637.24         0.00 
         OPS        16479     16089.04         0.00 
         OPS        23953     23439.02         0.00 
         OPS        15652     15187.21         0.00 
         OPS        17747     17217.91         0.00 

====== Total Duration 60.099 sec ======

Total Processes:          1
Total Threads:            1
Total Async Xacts:      400
Target TPS:               0
Total Xacts:        1138387
Total Conflicts:          0
Total Errors:             0
Total Timeouts:           0
Overall TPS:          18941

%CPU Worker Processes:          18.70 
%CPU Worker Threads:             3.34 
%CPU Local Network Threads:     14.86 
%CPU External Network Threads:   0.49 

                    CLEAR       COMMIT          TPS  Conflicts/s 
============ ============ ============ ============ ============ 
Total OPS         1138387      1138387     18941.73         0.00 
Errors                  0            0 
Timeouts                0            0 

Latency (us)       CLEAR       COMMIT  TRANSACTION 
============ ============ ============ ============ 
Samples              1200         1200         1200 
Min                     0        15093        15097 
Avg                     0        29392        29945 
Max                   N/A        46104        46124 
Median                  0        27021        28691 
95.0 pctile             0        27021        28691 
99.0 pctile             0        27021        28691 
99.9 pctile             0        27021        28691 

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

foundationdb-ci commented 1 month ago

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented 1 month ago

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

foundationdb-ci commented 1 month ago

Result of foundationdb-pr-macos on macOS Ventura 13.x

foundationdb-ci commented 1 month ago

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

foundationdb-ci commented 1 month ago

Result of foundationdb-pr-clang on Linux CentOS 7

foundationdb-ci commented 1 month ago

Result of foundationdb-pr on Linux CentOS 7

jzhou77 commented 1 month ago

100k 20240531-235953-jzhou-ae0ca5b29c7337b0