search

microsoft / CCF

Confidential Consortium Framework
https://microsoft.github.io/CCF/
Apache License 2.0
761 stars 205 forks source link

Consistency Spec Trace Validation #6124

Closed achamayou closed 4 weeks ago

achamayou commented 1 month ago

Early draft follow up to https://github.com/microsoft/CCF/pull/6116, same as #6119 but as a branch so the TLAi CI can access to token and run.

There is one known problem at the moment: there are other transactions in a real CCF, and so the txids aren't nicely contiguous as expected by the spec. @heidihoward had a solution to this, adding stub transactions and leveraging non-determinism, but I can't seem to find it right now.

The goal for this PR is to get this running and passing in CI for a prefix. It can be extended subsequently.

ccf-bot commented 1 month ago

consistency_tv@83712 aka 20240415.24 vs main ewma over 20 builds from 83365 to 83698

Click to see table main | build_id | build_number | tlc_sim_traces | tlc_sim_levelmean | Commit latency factor | tpcc_virtual_cft^ | ls_virtual_cft^ | pi_ls_virtual_cft^ | pi_basic_virtual_cft^ | pi_basic_js_virtual_cft^ | ls_jwt_virtual_cft^ | pi_ls_jwt_virtual_cft^ | ls_js_virtual_cft^ | ls_full_js_virtual_cft^ | ls_js_jwt_virtual_cft^ | hist_sgx_cft^ | pi_basic_mt_virtual_cft^ | tpcc_sgx_cft^ | tpcc_sgx_cft_mem | pi_basic_mt_sgx_cft^ | pi_basic_mt_sgx_cft_mem | ls_sgx_cft^ | ls_sgx_cft_mem | pi_ls_sgx_cft^ | pi_ls_sgx_cft_mem | pi_basic_sgx_cft^ | pi_basic_sgx_cft_mem | pi_basic_js_sgx_cft^ | pi_basic_js_sgx_cft_mem | ls_jwt_sgx_cft^ | ls_jwt_sgx_cft_mem | pi_ls_jwt_sgx_cft^ | pi_ls_jwt_sgx_cft_mem | ls_js_sgx_cft^ | ls_js_sgx_cft_mem | ls_full_js_sgx_cft^ | ls_full_js_sgx_cft_mem | ls_js_jwt_sgx_cft^ | ls_js_jwt_sgx_cft_mem | KV ser (/s)^ | KV deser (/s)^ | KV snap ser (/s)^ | KV snap deser (/s)^ | RB put (/s)^ | CHAMP put (/s)^ | RB get (/s)^ | CHAMP get (/s)^ | |-----------:|:---------------|-----------------:|--------------------:|------------------------:|--------------------:|------------------:|---------------------:|------------------------:|---------------------------:|----------------------:|-------------------------:|---------------------:|--------------------------:|-------------------------:|----------------:|---------------------------:|----------------:|-------------------:|-----------------------:|--------------------------:|--------------:|-----------------:|-----------------:|--------------------:|--------------------:|-----------------------:|-----------------------:|--------------------------:|------------------:|---------------------:|---------------------:|------------------------:|-----------------:|--------------------:|----------------------:|-------------------------:|---------------------:|------------------------:|-----------------:|-----------------:|--------------------:|----------------------:|---------------:|------------------:|---------------:|------------------:| | 83365 | 20240401.3 | 30754 | 377 | 0.810359 | 17195.5 | 52968 | 56448.1 | 60082.8 | 4417.7 | 20796.1 | 21019.9 | 17474.3 | 17492.9 | 11727.7 | 43823.7 | 68259.9 | 5627.14 | 8.59996e+07 | 27776.4 | 2.51822e+07 | 13873.6 | 1.88908e+07 | 13923.8 | 1.05021e+07 | 15440.9 | 1.46964e+07 | 1361.7 | 1.25993e+07 | 6842.56 | 1.88908e+07 | 6960.2 | 6.30784e+06 | 5810.32 | 1.67936e+07 | 5476.46 | 1.67936e+07 | 3997.84 | 1.67936e+07 | 1.02051e+06 | 1.12499e+06 | 7750.24 | 1401.2 | 834581 | 1.17926e+06 | 8.14907e+06 | 3.2663e+07 | | 83376 | 20240402.3 | 29810 | 383 | 0.799042 | 17267.7 | 52955.6 | 56862.8 | 59519.2 | 4425.2 | 20973.4 | 21697.2 | 17487.8 | 17373.7 | 11609.4 | 42110.8 | 83080.8 | 5555.59 | 8.59996e+07 | 27361.3 | 2.30851e+07 | 13917 | 1.88908e+07 | 13907.4 | 1.05021e+07 | 15383.4 | 1.46964e+07 | 1360.3 | 1.25993e+07 | 6825.59 | 1.88908e+07 | 6927.1 | 6.30784e+06 | 5807.65 | 1.67936e+07 | 5487.31 | 1.67936e+07 | 4008.13 | 1.67936e+07 | 1.01843e+06 | 1.15929e+06 | 7701.94 | 1408.13 | 832051 | 1.18464e+06 | 8.12041e+06 | 3.08708e+07 | | 83387 | 20240402.7 | 29554 | 372 | 0.818038 | 17383.2 | 52921.2 | 56391.1 | 61114.2 | 4398.2 | 20676 | 20801.7 | 17423.3 | 17097.1 | 11687.5 | 46105.2 | 88136.4 | 5579.52 | 8.59996e+07 | 27679.7 | 2.51822e+07 | 13822 | 1.88908e+07 | 13861.3 | 1.05021e+07 | 15353.6 | 1.46964e+07 | 1366.2 | 1.25993e+07 | 6873.64 | 1.88908e+07 | 6885.7 | 6.30784e+06 | 5799.29 | 1.67936e+07 | 5470.11 | 1.67936e+07 | 3964.49 | 1.67936e+07 | 1.04178e+06 | 1.16239e+06 | 7986.02 | 1403.94 | 828227 | 1.17352e+06 | 8.15313e+06 | 3.17092e+07 | | 83394 | 20240403.3 | 30720 | 376 | 0.795653 | 17334.1 | 53089.2 | 56972.9 | 61037.5 | 4413.7 | 20606.5 | 20959.1 | 17670.3 | 17377.7 | 11765.2 | 44785.5 | 69002.8 | 5570.57 | 8.59996e+07 | 27512.2 | 2.51822e+07 | 13917.8 | 1.88908e+07 | 13938.1 | 1.05021e+07 | 15484.8 | 1.46964e+07 | 1369.5 | 1.25993e+07 | 7186.56 | 1.67936e+07 | 6957.1 | 6.30784e+06 | 5802.37 | 1.67936e+07 | 5479.51 | 1.67936e+07 | 3995.27 | 1.67936e+07 | 1.03455e+06 | 1.17966e+06 | 7801.67 | 1406.73 | 833321 | 1.18091e+06 | 8.14314e+06 | 3.07166e+07 | | 83408 | 20240404.3 | 30391 | 379 | 0.836361 | 17318.8 | 52945.7 | 56267 | 62433.5 | 4456.1 | 21107 | 21717.9 | 17490.4 | 17251.1 | 11722.5 | 39727.3 | 96525.6 | 5617.23 | 8.59996e+07 | 27743 | 2.30851e+07 | 13941.1 | 1.88908e+07 | 14037.1 | 1.05021e+07 | 15562.4 | 1.46964e+07 | 1363.3 | 1.25993e+07 | 6903.33 | 1.88908e+07 | 6887.4 | 6.30784e+06 | 5799.01 | 1.67936e+07 | 5463.66 | 1.67936e+07 | 3987.88 | 1.67936e+07 | 1.02987e+06 | 1.13404e+06 | 7745.13 | 1402.62 | 831256 | 1.17788e+06 | 8.13557e+06 | 3.07554e+07 | | 83419 | 20240404.5 | 32428 | 382 | 0.821804 | 17256.1 | 52901.3 | 56589.6 | 61906 | 4403.3 | 20590.2 | 21153.3 | 17442.8 | 17106.5 | 11674.1 | 45979 | 69081.1 | 5577.86 | 8.59996e+07 | 27958.6 | 2.30851e+07 | 13973.3 | 1.67936e+07 | 13977.4 | 1.05021e+07 | 15377.4 | 1.46964e+07 | 1365.9 | 1.25993e+07 | 6825.56 | 1.88908e+07 | 6917.7 | 6.30784e+06 | 5803.51 | 1.67936e+07 | 5458.29 | 1.67936e+07 | 3801.64 | 1.67936e+07 | 1.00847e+06 | 1.12562e+06 | 7474.96 | 1398.42 | 845103 | 1.17785e+06 | 8.13509e+06 | 3.11663e+07 | | 83433 | 20240405.2 | 30697 | 373 | 0.795203 | 17432.5 | 53095.5 | 55527.2 | 61761.9 | 4429 | 20818 | 21148.8 | 17215.2 | 17096.2 | 11509 | 44641.6 | 73717.7 | 5610.51 | 8.59996e+07 | 27805.9 | 2.30851e+07 | 13933.8 | 1.67936e+07 | 14008.4 | 1.05021e+07 | 15466.9 | 1.25993e+07 | 1360.5 | 1.25993e+07 | 6820.03 | 1.88908e+07 | 6932.1 | 6.30784e+06 | 5755.14 | 1.67936e+07 | 5461.46 | 1.67936e+07 | 3987.78 | 1.67936e+07 | 1.0175e+06 | 1.18779e+06 | 8272.95 | 1408.52 | 837389 | 1.1752e+06 | 8.15196e+06 | 3.06927e+07 | | 83446 | 20240408.2 | 31171 | 383 | 0.80954 | 17447.2 | 53117.1 | 55958.8 | 61153.3 | 4432.1 | 20761.9 | 20873.8 | 17457.9 | 17374.6 | 11529.9 | 47154.6 | 91917.1 | 5516.27 | 8.59996e+07 | 27601.3 | 2.30851e+07 | 13896.2 | 1.88908e+07 | 13970.4 | 1.05021e+07 | 15383.1 | 1.46964e+07 | 1363 | 1.25993e+07 | 6825.46 | 1.88908e+07 | 6988.6 | 6.30784e+06 | 5759.72 | 1.67936e+07 | 5425.1 | 1.67936e+07 | 3992.3 | 1.67936e+07 | 1.02533e+06 | 1.10779e+06 | 7833.59 | 1393.37 | 813419 | 1.18481e+06 | 8.14959e+06 | 3.15227e+07 | | 83465 | 20240409.3 | 31097 | 380 | 0.819344 | 17275.2 | 53026.3 | 56413.7 | 62184.1 | 4436.4 | 20560.2 | 21632.4 | 17479.3 | 17314.9 | 11682.1 | 45530.7 | 93716.1 | 5595.61 | 8.59996e+07 | 27515.3 | 2.30851e+07 | 13937.9 | 1.67936e+07 | 13959.8 | 1.05021e+07 | 15443.9 | 1.46964e+07 | 1360.5 | 1.25993e+07 | 6888.84 | 1.88908e+07 | 7084.5 | 6.30784e+06 | 5785.52 | 1.67936e+07 | 5456.19 | 1.67936e+07 | 3964.71 | 1.67936e+07 | 1.05053e+06 | 1.13559e+06 | 7700.05 | 1398.41 | 839459 | 1.18433e+06 | 8.15345e+06 | 3.08211e+07 | | 83491 | 20240409.11 | 29923 | 371 | 0.788667 | 17216.4 | 52730.3 | 57049 | 60114 | 4376.3 | 20863.8 | 22315 | 17644.1 | 17141.5 | 11680.7 | 43435.3 | 95032.1 | 5577.9 | 8.59996e+07 | 27923.2 | 2.51822e+07 | 13915.5 | 1.88908e+07 | 13948.1 | 1.05021e+07 | 15257.7 | 1.46964e+07 | 1350.5 | 1.25993e+07 | 6833.2 | 1.88908e+07 | 6936.2 | 6.30784e+06 | 5782.62 | 1.67936e+07 | 5459.02 | 1.67936e+07 | 3807.6 | 1.67936e+07 | 1.01225e+06 | 1.15969e+06 | 7776.73 | 1406.33 | 833597 | 1.17995e+06 | 8.15121e+06 | 3.15465e+07 | | 83516 | 20240410.3 | 31688 | 378 | 0.820327 | 17408.1 | 52988.5 | 55669.4 | 58564.1 | 4341.5 | 20881.9 | 22172 | 17604.5 | 17131.6 | 11516.6 | 46259.3 | 77993.3 | 5559.4 | 8.59996e+07 | 27661 | 2.51822e+07 | 13897 | 1.88908e+07 | 13983.7 | 1.05021e+07 | 15330.8 | 1.46964e+07 | 1365.5 | 1.05021e+07 | 7191.21 | 1.67936e+07 | 6872.9 | 6.30784e+06 | 5784.58 | 1.67936e+07 | 5461.12 | 1.67936e+07 | 3974.35 | 1.67936e+07 | 1.04712e+06 | 1.11433e+06 | 7652.8 | 1399.18 | 829881 | 1.18481e+06 | 8.13531e+06 | 3.0748e+07 | | 83524 | 20240410.5 | 31607 | 384 | 0.802161 | 17232.4 | 52988.3 | 57020.7 | 60754 | 4424.8 | 17578.5 | 21873.2 | 17603.5 | 17103.2 | 11561.1 | 44620.9 | 77062.9 | 5566.3 | 8.59996e+07 | 27560.8 | 2.51822e+07 | 13947.6 | 1.67936e+07 | 14044.3 | 1.05021e+07 | 15463.8 | 1.46964e+07 | 1370.2 | 1.25993e+07 | 7229.68 | 1.67936e+07 | 6998.4 | 6.30784e+06 | 5759.07 | 1.67936e+07 | 5469.21 | 1.67936e+07 | 3982.97 | 1.67936e+07 | 1.03918e+06 | 1.17661e+06 | 8224.64 | 1408.31 | 839904 | 1.17574e+06 | 8.13738e+06 | 3.08834e+07 | | 83548 | 20240411.2 | 29736 | 377 | 0.824138 | 17210.8 | 52940.3 | 56584.8 | 61285.2 | 4401.1 | 20862.1 | 21969.8 | 17551.3 | 17283 | 11494.4 | 43389.8 | 66739 | 5607.15 | 8.59996e+07 | 27689.7 | 2.51822e+07 | 13939.6 | 1.67936e+07 | 14071.7 | 1.05021e+07 | 15436.3 | 1.25993e+07 | 1366.1 | 1.25993e+07 | 6841.5 | 1.88908e+07 | 6878.9 | 6.30784e+06 | 5790.47 | 1.67936e+07 | 5457.03 | 1.67936e+07 | 3958.92 | 1.67936e+07 | 1.04297e+06 | 1.1722e+06 | 7930.42 | 1400.58 | 836833 | 1.17803e+06 | 8.15017e+06 | 3.07651e+07 | | 83561 | 20240411.8 | 31146 | 370 | 0.812126 | 17310.1 | 53142.9 | 55831.9 | 60447.1 | 4426.8 | 20937.9 | 22065.3 | 17587.6 | 17268.9 | 11657 | 42451.7 | 84111.9 | 5513.17 | 8.59996e+07 | 27419.8 | 2.30851e+07 | 13991.3 | 1.88908e+07 | 14012.3 | 1.05021e+07 | 15256 | 1.46964e+07 | 1361.1 | 1.25993e+07 | 7208.84 | 1.67936e+07 | 7010.4 | 6.30784e+06 | 5757.92 | 1.67936e+07 | 5485.1 | 1.67936e+07 | 3967.59 | 1.67936e+07 | 1.01874e+06 | 1.14626e+06 | 7628.61 | 1397.55 | 834446 | 1.17887e+06 | 8.15429e+06 | 3.11701e+07 | | 83575 | 20240411.14 | 30477 | 384 | 0.810011 | 16816.1 | 53136.1 | 55252.9 | 60586.4 | 4364.3 | 20640.6 | 21555 | 17274.5 | 17973.1 | 11534.2 | 39578.1 | 70051.8 | 5541.86 | 8.59996e+07 | 27574.4 | 2.30851e+07 | 13851.7 | 1.88908e+07 | 13944 | 1.05021e+07 | 15145.1 | 1.46964e+07 | 1357.7 | 1.25993e+07 | 6808.77 | 1.67936e+07 | 6877.6 | 6.30784e+06 | 5738.61 | 1.67936e+07 | 5427.42 | 1.67936e+07 | 3805.11 | 1.67936e+07 | 1.01461e+06 | 1.14456e+06 | 7640.76 | 1403.34 | 838300 | 1.17792e+06 | 8.13444e+06 | 3.06532e+07 | | 83594 | 20240412.2 | 30690 | 374 | 0.778222 | 17471.7 | 53108.6 | 56340.9 | 61266.1 | 4433.7 | 20692.3 | 22185.7 | 17141.4 | 17139.9 | 11542 | 38586 | 70582.5 | 5633.59 | 8.59996e+07 | 27746.4 | 2.51822e+07 | 13909.6 | 1.88908e+07 | 14038.6 | 1.05021e+07 | 15425.7 | 1.25993e+07 | 1362.8 | 1.25993e+07 | 7217.68 | 1.67936e+07 | 6868.1 | 6.30784e+06 | 5790.23 | 1.67936e+07 | 5452.72 | 1.67936e+07 | 3989.98 | 1.67936e+07 | 997904 | 1.15194e+06 | 7926.93 | 1404.25 | 836820 | 1.17884e+06 | 8.15524e+06 | 3.065e+07 | | 83618 | 20240412.12 | 32407 | 368 | 0.790379 | 17440.9 | 53138.7 | 55688.6 | 61148.6 | 4398.3 | 20639.7 | 21416.9 | 17071.7 | 17528.5 | 11430.4 | 43299.8 | 88740.9 | 5607.27 | 8.59996e+07 | 27694.6 | 2.51822e+07 | 13919 | 1.88908e+07 | 14023.5 | 1.05021e+07 | 15491.4 | 1.46964e+07 | 1372.1 | 1.05021e+07 | 6809 | 1.67936e+07 | 7039.8 | 6.30784e+06 | 5770.46 | 1.67936e+07 | 5453.33 | 1.67936e+07 | 3977.28 | 1.67936e+07 | 1.03638e+06 | 1.19332e+06 | 7840.6 | 1406.97 | 837854 | 1.18407e+06 | 8.15231e+06 | 3.09581e+07 | | 83641 | 20240415.3 | 29478 | 385 | 0.776887 | 17323.7 | 53050.4 | 56883.5 | 62579.3 | 4377.5 | 21214.8 | 21756.6 | 17224.3 | 17612.2 | 11429.6 | 45669.7 | 93199.7 | 5627.33 | 8.59996e+07 | 27523.6 | 2.30851e+07 | 13964.3 | 1.67936e+07 | 14023.2 | 1.05021e+07 | 15473.5 | 1.46964e+07 | 1360.1 | 1.25993e+07 | 6848.25 | 1.67936e+07 | 6934.3 | 6.30784e+06 | 5785.3 | 1.67936e+07 | 5451.5 | 1.67936e+07 | 3795.38 | 1.67936e+07 | 997208 | 1.14116e+06 | 8362.53 | 1404.68 | 835227 | 1.18569e+06 | 8.15387e+06 | 3.07623e+07 | | 83644 | 20240415.4 | 31602 | 377 | 0.810067 | 17286.2 | 52799.1 | 55584.3 | 61537.7 | 4390.7 | 20660.8 | 22135.6 | 17636.8 | 17686.6 | 11565.7 | 43965.9 | 95534.6 | 5600.24 | 8.59996e+07 | 27870.4 | 2.51822e+07 | 13937.9 | 1.88908e+07 | 14059.6 | 1.05021e+07 | 15481.2 | 1.25993e+07 | 1365.2 | 1.25993e+07 | 7192.76 | 1.67936e+07 | 6866.4 | 6.30784e+06 | 5760.09 | 1.67936e+07 | 5414.46 | 1.67936e+07 | 3938.89 | 1.67936e+07 | 1.03896e+06 | 1.12994e+06 | 7868.03 | 1401.74 | 839603 | 1.18e+06 | 8.17519e+06 | 3.08178e+07 | | 83698 | 20240415.20 | 31129 | 388 | 0.777621 | 17227.9 | 52976.3 | 56041.9 | 62006 | 4387.1 | 20981.1 | 21832.2 | 17235.5 | 17447.4 | 11400.1 | 43589.4 | 67929.5 | 5593.97 | 8.59996e+07 | 27490.4 | 2.30851e+07 | 13922.5 | 1.88908e+07 | 14040.3 | 1.05021e+07 | 15501.7 | 1.25993e+07 | 1369.3 | 1.05021e+07 | 6800.92 | 1.67936e+07 | 7031.2 | 6.30784e+06 | 5733.35 | 1.67936e+07 | 5435.7 | 1.67936e+07 | 3964.27 | 1.67936e+07 | 1.01297e+06 | 1.18287e+06 | 8254.33 | 1401.83 | 830349 | 1.1788e+06 | 8.11015e+06 | 3.07512e+07 | consistency_tv | build_id | build_number | Commit latency factor | tpcc_sgx_cft^ | tpcc_sgx_cft_mem | ls_sgx_cft^ | ls_sgx_cft_mem | pi_ls_sgx_cft^ | pi_ls_sgx_cft_mem | pi_basic_sgx_cft^ | pi_basic_sgx_cft_mem | pi_basic_js_sgx_cft^ | pi_basic_js_sgx_cft_mem | ls_jwt_sgx_cft^ | ls_jwt_sgx_cft_mem | pi_ls_jwt_sgx_cft^ | pi_ls_jwt_sgx_cft_mem | ls_js_sgx_cft^ | ls_js_sgx_cft_mem | ls_full_js_sgx_cft^ | ls_full_js_sgx_cft_mem | ls_js_jwt_sgx_cft^ | ls_js_jwt_sgx_cft_mem | hist_sgx_cft^ | KV ser (/s)^ | KV deser (/s)^ | KV snap ser (/s)^ | KV snap deser (/s)^ | RB put (/s)^ | CHAMP put (/s)^ | RB get (/s)^ | CHAMP get (/s)^ | pi_basic_mt_sgx_cft^ | pi_basic_mt_sgx_cft_mem | tlc_sim_traces | tlc_sim_levelmean | pi_basic_mt_virtual_cft^ | tpcc_virtual_cft^ | ls_virtual_cft^ | pi_ls_virtual_cft^ | pi_basic_virtual_cft^ | pi_basic_js_virtual_cft^ | ls_jwt_virtual_cft^ | pi_ls_jwt_virtual_cft^ | ls_js_virtual_cft^ | ls_full_js_virtual_cft^ | ls_js_jwt_virtual_cft^ | |-----------:|:---------------|------------------------:|----------------:|-------------------:|--------------:|-----------------:|-----------------:|--------------------:|--------------------:|-----------------------:|-----------------------:|--------------------------:|------------------:|---------------------:|---------------------:|------------------------:|-----------------:|--------------------:|----------------------:|-------------------------:|---------------------:|------------------------:|----------------:|---------------:|-----------------:|--------------------:|----------------------:|---------------:|------------------:|---------------:|------------------:|-----------------------:|--------------------------:|-----------------:|--------------------:|---------------------------:|--------------------:|------------------:|---------------------:|------------------------:|---------------------------:|----------------------:|-------------------------:|---------------------:|--------------------------:|-------------------------:| | 83712 | 20240415.24 | 0.813137 | 5516.57 | 8.39025e+07 | 13922.6 | 1.88908e+07 | 13974.7 | 1.05021e+07 | 15343.6 | 1.25993e+07 | 1344.9 | 1.25993e+07 | 6763.77 | 1.88908e+07 | 6844.7 | 6.30784e+06 | 5481.28 | 1.67936e+07 | 5438.81 | 1.67936e+07 | 3790.74 | 1.67936e+07 | 45554.7 | 1.03498e+06 | 1.15141e+06 | 8036.78 | 1403.96 | 838235 | 1.1827e+06 | 8.1518e+06 | 3.08489e+07 | 27436.5 | 2.30851e+07 | 30827 | 382 | 94743.8 | 17321 | 53126 | 55928.1 | 62443.7 | 4306.5 | 20792.9 | 22135.2 | 17321.9 | 17695.6 | 11429.7 |

images