PR of the perf tool generator component

[fotiskoun]

This is the generator component of the performance tool. It includes a file containing the functions for the construction of the parquet output, a file for command line execution, and a sample file using the functions for the creation of more complex scenarios.

[achamayou]

@MahatiC this is the generator part of #848

[heidihoward]

@MahatiC just to elaborate on @achamayou's response.

This RP contains the first of three components (generator, submitter & analyzer) of a new load testing tool for CCF that @fotiskoun is building. The plans for this tool are described in https://github.com/microsoft/CCF/discussions/2324 & https://github.com/microsoft/CCF/issues/848. @fotiskoun has been working so far on a branch of CCF https://github.com/fotiskoun/CCF/tree/testing-performance-system and this is the first of a few PRs to merge his efforts with CCF mainline. He has split the contributions into smaller RP for easier review, but this does mean that this first PR is not directly useful on its own. The generator library & cli produces parquet files that will be the input to the submitter component.

[MahatiC]

@heidihoward Thank you for the background and details!

[heidihoward]

@MahatiC No worries!

Please let us know how we can better document this tool. For instance, we could create a page in the docs if that helps?

[MahatiC]

For instance, we could create a page in the docs if that helps?

@heidihoward I think once we have all the components, we could then have a doc page explaining the perf system.

[achamayou]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 2 pipeline(s).

[ccf-bot]

generator-pr@52992 aka 20221102.15 vs main ewma over 20 builds from 52529 to 52978

Click to see table

main | build_id | build_number | tpcc_virtual_cft^ | ls_virtual_cft^ | ls_jwt_virtual_cft^ | ls_js_virtual_cft^ | ls_full_js_virtual_cft^ | ls_js_jwt_virtual_cft^ | hist_sgx_cft^ | tpcc_sgx_cft^ | tpcc_sgx_cft_mem | ls_sgx_cft^ | ls_sgx_cft_mem | ls_jwt_sgx_cft^ | ls_jwt_sgx_cft_mem | ls_js_sgx_cft^ | ls_js_sgx_cft_mem | ls_full_js_sgx_cft^ | ls_full_js_sgx_cft_mem | ls_js_jwt_sgx_cft^ | ls_js_jwt_sgx_cft_mem | RB put (/s)^ | CHAMP put (/s)^ | RB get (/s)^ | CHAMP get (/s)^ | Commit latency factor | |-----------:|:---------------|--------------------:|------------------:|----------------------:|---------------------:|--------------------------:|-------------------------:|----------------:|----------------:|-------------------:|--------------:|-----------------:|------------------:|---------------------:|-----------------:|--------------------:|----------------------:|-------------------------:|---------------------:|------------------------:|---------------:|------------------:|---------------:|------------------:|------------------------:| | 52529 | 20221031.1 | 11317.6 | 42713.1 | 10189.6 | 4384.6 | 3465.19 | 3239.39 | 28320.2 | 5554.34 | 8.32349e+07 | 17385.6 | 1.6126e+07 | 5952.21 | 1.53396e+07 | 2353.33 | 9.83456e+06 | 1949.8 | 9.57242e+06 | 1887.19 | 9.04813e+06 | 889422 | 1.38191e+06 | 9.39441e+06 | 3.58036e+07 | nan | | 52542 | 20221031.6 | 11057.3 | 40617.7 | 9983.96 | 4282.55 | 3477.62 | 3205.73 | 22787.8 | 5640.13 | 8.32349e+07 | 17419.4 | 1.63882e+07 | 5984.67 | 1.56017e+07 | 2359.4 | 9.83456e+06 | 1982.98 | 9.57242e+06 | 1915.42 | 9.31027e+06 | 886345 | 1.36215e+06 | 9.24184e+06 | 3.58036e+07 | nan | | 52577 | 20221031.16 | 11210.4 | 41561.5 | 10057.5 | 4133.37 | 3422.54 | 3247.85 | 24348.4 | 5682.73 | 8.29727e+07 | 17454.3 | 1.66503e+07 | 6017.69 | 1.53396e+07 | 2361.67 | 9.83456e+06 | 1989.35 | 1.19317e+07 | 1926.84 | 9.04813e+06 | 901918 | 1.36441e+06 | 9.21688e+06 | 3.58663e+07 | nan | | 52627 | 20221031.31 | 10389.6 | 36465.4 | 10643.7 | 4070.62 | 3376.38 | 3209.04 | 25417.3 | 5461.8 | 8.24484e+07 | 15920.2 | 1.6126e+07 | 5469.97 | 1.50774e+07 | 2049.2 | 8.52384e+06 | 1679.1 | 9.04813e+06 | 1616.37 | 7.99955e+06 | 831354 | 1.17052e+06 | 8.15167e+06 | 3.08058e+07 | nan | | 52629 | 20221031.32 | 11537.3 | 42144.4 | 9966.78 | 4325.53 | 3326.76 | 3245.11 | 23577.6 | 5227.29 | 8.32349e+07 | 15549 | 1.63882e+07 | 5344.77 | 1.56017e+07 | 2033.02 | 8.52384e+06 | 1671.56 | 9.04813e+06 | 1581.75 | 7.99955e+06 | 839644 | 1.17704e+06 | 8.15381e+06 | 3.11544e+07 | nan | | 52633 | 20221101.2 | 10922.3 | 40768.4 | 10729.3 | 4232.88 | 3441.42 | 3245.17 | 21332.1 | 5157.48 | 8.27106e+07 | 15554.8 | 1.63882e+07 | 5361.14 | 1.53396e+07 | 1998.28 | 8.78598e+06 | 1665.53 | 9.04813e+06 | 1585.35 | 7.99955e+06 | 832231 | 1.181e+06 | 8.17493e+06 | 3.13629e+07 | nan | | 52718 | 20221101.29 | 10978.6 | 40067.5 | 10392.9 | 4050.44 | 3419.23 | 3337.06 | 25803.1 | 5534.62 | 8.29727e+07 | 16197.8 | 1.6126e+07 | 5473.76 | 1.53396e+07 | 2046.41 | 8.78598e+06 | 1698.91 | 8.78598e+06 | 1637.36 | 7.99955e+06 | 841238 | 1.18106e+06 | 8.15566e+06 | 3.12958e+07 | nan | | 52744 | 20221101.38 | 11456.4 | 41041.3 | 10209.1 | 4299.1 | 3489.45 | 3234.76 | 21213.2 | 5502.32 | 8.32349e+07 | 16097.8 | 1.63882e+07 | 5445.07 | 1.56017e+07 | 2031.6 | 8.78598e+06 | 1683.42 | 8.78598e+06 | 1616.71 | 8.2617e+06 | 836979 | 1.17435e+06 | 8.15754e+06 | 3.0842e+07 | nan | | 52774 | 20221101.47 | 10552.1 | 41164.6 | 10321.5 | 4279.33 | 3399.1 | 3125.67 | 25082.2 | 5585.58 | 8.27106e+07 | 16217.2 | 1.63882e+07 | 5417.87 | 1.53396e+07 | 2038.42 | 8.52384e+06 | 1691.13 | 8.52384e+06 | 1626.78 | 7.73741e+06 | 837070 | 1.18686e+06 | 8.17147e+06 | 3.11218e+07 | nan | | 52809 | 20221101.56 | 11354.1 | 40616.9 | 10066.3 | 4275.94 | 3714.35 | 3269.29 | 26058.8 | 5483.38 | 8.29727e+07 | 15754.8 | 1.6126e+07 | 5392.28 | 1.53396e+07 | 2040.53 | 8.52384e+06 | 1692.91 | 8.52384e+06 | 1637.8 | 7.99955e+06 | 835548 | 1.17751e+06 | 8.14839e+06 | 3.08248e+07 | nan | | 52836 | 20221101.65 | 12975.4 | 38685.7 | 11022.1 | 3904.35 | 3125.32 | 2872.24 | 22079.4 | 5666.78 | 8.3497e+07 | 15879.2 | 1.53396e+07 | 5692.48 | 1.45532e+07 | 2058.98 | 9.04813e+06 | 1682.43 | 8.78598e+06 | 1602.04 | 7.99955e+06 | 838363 | 1.18024e+06 | 8.13493e+06 | 3.07111e+07 | 0.840398 | | 52860 | 20221101.73 | 16788.9 | 45576.9 | 12595.4 | 4556.62 | 3453.83 | 3222.78 | 25456.5 | 6273.01 | 8.21863e+07 | 16616.8 | 1.53396e+07 | 5956.18 | 1.50774e+07 | 2073.62 | 8.78598e+06 | 1689.81 | 8.52384e+06 | 1625.04 | 7.73741e+06 | 829373 | 1.17726e+06 | 8.17425e+06 | 3.14656e+07 | 0.768566 | | 52889 | 20221101.81 | 17527.2 | 43805.5 | 12626.2 | 4401.04 | 3588.18 | 3362.47 | 29637 | 6306.08 | 8.29727e+07 | 16253.5 | 1.50774e+07 | 5941.91 | 1.50774e+07 | 2069.39 | 8.52384e+06 | 1702.49 | 8.78598e+06 | 1623.71 | 7.99955e+06 | 839729 | 1.17486e+06 | 8.15592e+06 | 3.06959e+07 | 0.782895 | | 52903 | 20221101.86 | 17537.9 | 43610.3 | 12588.6 | 4643.07 | 3561.49 | 3229.48 | 21617.8 | 5669.25 | 8.27106e+07 | 16013.7 | 1.56017e+07 | 5655.71 | 1.48153e+07 | 2030.98 | 8.52384e+06 | 1676.91 | 8.52384e+06 | 1579.98 | 7.99955e+06 | 832062 | 1.17538e+06 | 8.16971e+06 | 3.08824e+07 | 0.815599 | | 52919 | 20221101.90 | 17265.2 | 45770.4 | 12606.2 | 4582.28 | 3482.84 | 3361.88 | 24079.5 | 5588.57 | 8.32349e+07 | 16023.3 | 1.58639e+07 | 5721.75 | 1.48153e+07 | 2029.63 | 8.78598e+06 | 1684.49 | 9.04813e+06 | 1613.61 | 7.99955e+06 | 838236 | 1.1826e+06 | 8.15374e+06 | 3.08703e+07 | 0.826385 | | 52928 | 20221101.94 | 17061.2 | 41901.6 | 12714.4 | 4566.57 | 3623.15 | 3371.65 | 24093.5 | 5580.58 | 8.27106e+07 | 16062.6 | 1.53396e+07 | 5905.59 | 1.45532e+07 | 2035.71 | 8.78598e+06 | 1676.17 | 8.78598e+06 | 1608.74 | 7.99955e+06 | 837289 | 1.17643e+06 | 8.17102e+06 | 3.07364e+07 | 0.816214 | | 52934 | 20221101.96 | 17564.9 | 43657.7 | 12483.7 | 4369.81 | 3604.1 | 3374.04 | 23964.5 | 6266.01 | 8.19242e+07 | 16478.7 | 1.50774e+07 | 5868.9 | 1.45532e+07 | 2059.88 | 8.52384e+06 | 1673.54 | 8.78598e+06 | 1635.75 | 7.73741e+06 | 839550 | 1.17918e+06 | 8.16528e+06 | 3.11417e+07 | 0.761476 | | 52941 | 20221101.98 | 17782.2 | 43502.2 | 12638.7 | 4433.67 | 3454.54 | 3246.07 | 27599.3 | 6388.18 | 8.21863e+07 | 16628 | 1.50774e+07 | 5906.45 | 1.56017e+07 | 2099.33 | 9.31027e+06 | 1708.45 | 8.78598e+06 | 1641.61 | 7.99955e+06 | 839624 | 1.17998e+06 | 8.15316e+06 | 3.08941e+07 | 0.800341 | | 52956 | 20221102.2 | 17372.4 | 41820.9 | 12622.6 | 4320.99 | 3451.7 | 3372.48 | 27099.5 | 6382.1 | 8.21863e+07 | 16629.9 | 1.56017e+07 | 5919.66 | 1.50774e+07 | 2095.36 | 8.78598e+06 | 1707.65 | 8.78598e+06 | 1642.92 | 7.99955e+06 | 834176 | 1.18165e+06 | 8.15193e+06 | 3.15976e+07 | 0.810117 | | 52978 | 20221102.10 | 16900.8 | 43669.1 | 12356.4 | 4263.93 | 3446.79 | 3212.91 | 27567.5 | 6337.88 | 8.24484e+07 | 16612.4 | 1.50774e+07 | 5867.78 | 1.50774e+07 | 2101.14 | 8.78598e+06 | 1713.86 | 8.52384e+06 | 1651.38 | 7.99955e+06 | 829188 | 1.18143e+06 | 8.15231e+06 | 3.10247e+07 | 0.741081 | generator-pr | build_id | build_number | Commit latency factor | tpcc_virtual_cft^ | ls_virtual_cft^ | ls_jwt_virtual_cft^ | ls_js_virtual_cft^ | ls_full_js_virtual_cft^ | tpcc_sgx_cft^ | tpcc_sgx_cft_mem | ls_js_jwt_virtual_cft^ | ls_sgx_cft^ | ls_sgx_cft_mem | ls_jwt_sgx_cft^ | ls_jwt_sgx_cft_mem | hist_sgx_cft^ | ls_js_sgx_cft^ | ls_js_sgx_cft_mem | ls_full_js_sgx_cft^ | ls_full_js_sgx_cft_mem | ls_js_jwt_sgx_cft^ | ls_js_jwt_sgx_cft_mem | RB put (/s)^ | CHAMP put (/s)^ | RB get (/s)^ | CHAMP get (/s)^ | |-----------:|:---------------|------------------------:|--------------------:|------------------:|----------------------:|---------------------:|--------------------------:|----------------:|-------------------:|-------------------------:|--------------:|-----------------:|------------------:|---------------------:|----------------:|-----------------:|--------------------:|----------------------:|-------------------------:|---------------------:|------------------------:|---------------:|------------------:|---------------:|------------------:| | 52921 | 20221101.91 | 0.772328 | 17688.3 | 42051.8 | 12634.1 | 4579.06 | 3610.71 | 6288.51 | 8.24484e+07 | 3366.14 | 16565.8 | 1.50774e+07 | 5766.51 | 1.48153e+07 | 23985.4 | 2091.1 | 8.52384e+06 | 1707.85 | 8.52384e+06 | 1647.47 | 7.99955e+06 | 833041 | 1.173e+06 | 8.17242e+06 | 3.081e+07 | | 52961 | 20221102.4 | 0.778036 | 17296.5 | 45764.8 | 12573.6 | 4384.3 | 3548.58 | 6286.89 | 8.27106e+07 | 3337.2 | 16655.5 | 1.58639e+07 | 5833.11 | 1.50774e+07 | 21912.3 | 2101.92 | 8.52384e+06 | 1714.55 | 8.52384e+06 | 1650.51 | 8.2617e+06 | 842470 | 1.17962e+06 | 8.17483e+06 | 3.21694e+07 | | 52992 | 20221102.15 | 0.76565 | 17260.2 | 43729.9 | 12672.3 | 4573.65 | 3578.82 | 6306.23 | 8.27106e+07 | 3260.07 | 16628.3 | 1.50774e+07 | 5854.43 | 1.50774e+07 | 23190 | 2068.65 | 8.78598e+06 | 1713.04 | 8.52384e+06 | 1615.93 | 7.99955e+06 | 825105 | 1.17393e+06 | 8.1748e+06 | 3.12076e+07 |

[heidihoward]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 2 pipeline(s).

[heidihoward]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 2 pipeline(s).