Reduce benchmarking time limits

[kwxm]

This is a followup to #6328. Criterion runs each individual benchmark up to some time limit then analyses the times of the different runs. Our benchmark suites currently get run with different time limits: 60s for nofib, 20 for validation, and 15 for lists. There should be scope for reducing these limits without reducing the accuracy of the results too much. I'm going to do some experiments in this PR to see what happens. Using /benchmark, the benchmarks will be run in the base branch with the old time limits and in this branch with the new ones, so we should be able to get a decent comparison.

[kwxm]

/benchmark nofib

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

[Time limit = 30s]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | clausify/formula1 | 3.115 ms | 3.097 ms | -0.6% | | clausify/formula2 | 4.204 ms | 4.147 ms | -1.4% | | clausify/formula3 | 11.51 ms | 11.37 ms | -1.2% | | clausify/formula4 | 26.51 ms | 25.85 ms | -2.5% | | clausify/formula5 | 55.76 ms | 54.95 ms | -1.5% | | knights/4x4 | 18.33 ms | 17.79 ms | -2.9% | | knights/6x6 | 47.98 ms | 46.56 ms | -3.0% | | knights/8x8 | 84.09 ms | 81.46 ms | -3.1% | | primetest/05digits | 10.88 ms | 10.19 ms | -6.3% | | primetest/10digits | 21.24 ms | 19.86 ms | -6.5% | | primetest/30digits | 65.31 ms | 61.24 ms | -6.2% | | primetest/50digits | 108.4 ms | 101.7 ms | -6.2% | | queens4x4/bt | 5.514 ms | 5.400 ms | -2.1% | | queens4x4/bm | 6.975 ms | 6.851 ms | -1.8% | | queens4x4/bjbt1 | 6.702 ms | 6.547 ms | -2.3% | | queens4x4/bjbt2 | 6.279 ms | 6.096 ms | -2.9% | | queens4x4/fc | 14.13 ms | 13.52 ms | -4.3% | | queens5x5/bt | 74.80 ms | 71.40 ms | -4.5% | | queens5x5/bm | 77.96 ms | 75.00 ms | -3.8% | | queens5x5/bjbt1 | 87.51 ms | 83.48 ms | -4.6% | | queens5x5/bjbt2 | 84.97 ms | 81.94 ms | -3.6% | | queens5x5/fc | 179.8 ms | 174.9 ms | -2.7% | | | | | | | TOTAL | 1.002 s | 963.3 ms | -3.9% |

[kwxm]

/benchmark nofib

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | clausify/formula1 | 3.033 ms | 3.025 ms | -0.3% | | clausify/formula2 | 4.073 ms | 4.058 ms | -0.4% | | clausify/formula3 | 11.13 ms | 11.15 ms | +0.2% | | clausify/formula4 | 25.34 ms | 25.46 ms | +0.5% | | clausify/formula5 | 53.89 ms | 54.56 ms | +1.2% | | knights/4x4 | 17.47 ms | 17.59 ms | +0.7% | | knights/6x6 | 45.54 ms | 45.81 ms | +0.6% | | knights/8x8 | 79.83 ms | 80.81 ms | +1.2% | | primetest/05digits | 10.01 ms | 10.26 ms | +2.5% | | primetest/10digits | 19.60 ms | 20.09 ms | +2.5% | | primetest/30digits | 60.11 ms | 62.09 ms | +3.3% | | primetest/50digits | 100.5 ms | 103.5 ms | +3.0% | | queens4x4/bt | 5.269 ms | 5.341 ms | +1.4% | | queens4x4/bm | 6.721 ms | 6.806 ms | +1.3% | | queens4x4/bjbt1 | 6.416 ms | 6.508 ms | +1.4% | | queens4x4/bjbt2 | 6.010 ms | 6.095 ms | +1.4% | | queens4x4/fc | 13.56 ms | 13.75 ms | +1.4% | | queens5x5/bt | 71.65 ms | 72.85 ms | +1.7% | | queens5x5/bm | 75.22 ms | 76.54 ms | +1.8% | | queens5x5/bjbt1 | 84.21 ms | 85.37 ms | +1.4% | | queens5x5/bjbt2 | 81.56 ms | 83.05 ms | +1.8% | | queens5x5/fc | 170.9 ms | 174.5 ms | +2.1% | | | | | | | TOTAL | 952.0 ms | 969.2 ms | +1.8% |

[kwxm]

/benchmark nofib

[kwxm]

/benchmark validation

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | clausify/formula1 | 3.087 ms | 3.043 ms | -1.4% | | clausify/formula2 | 4.133 ms | 4.075 ms | -1.4% | | clausify/formula3 | 11.29 ms | 11.17 ms | -1.1% | | clausify/formula4 | 25.73 ms | 25.44 ms | -1.1% | | clausify/formula5 | 54.78 ms | 54.08 ms | -1.3% | | knights/4x4 | 17.73 ms | 17.69 ms | -0.2% | | knights/6x6 | 46.13 ms | 46.08 ms | -0.1% | | knights/8x8 | 80.80 ms | 80.50 ms | -0.4% | | primetest/05digits | 10.13 ms | 10.16 ms | +0.3% | | primetest/10digits | 19.77 ms | 19.89 ms | +0.6% | | primetest/30digits | 61.07 ms | 61.33 ms | +0.4% | | primetest/50digits | 102.0 ms | 102.0 ms | 0.0% | | queens4x4/bt | 5.296 ms | 5.321 ms | +0.5% | | queens4x4/bm | 6.752 ms | 6.791 ms | +0.6% | | queens4x4/bjbt1 | 6.451 ms | 6.480 ms | +0.4% | | queens4x4/bjbt2 | 6.036 ms | 6.067 ms | +0.5% | | queens4x4/fc | 13.62 ms | 13.70 ms | +0.6% | | queens5x5/bt | 71.94 ms | 72.45 ms | +0.7% | | queens5x5/bm | 75.47 ms | 76.07 ms | +0.8% | | queens5x5/bjbt1 | 83.95 ms | 84.76 ms | +1.0% | | queens5x5/bjbt2 | 81.52 ms | 82.12 ms | +0.7% | | queens5x5/fc | 172.4 ms | 173.6 ms | +0.7% | | | | | | | TOTAL | 960.1 ms | 962.8 ms | +0.3% |

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Comparing benchmark results of 'validation' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | auction_1-1 | 182.9 μs | 182.0 μs | -0.5% | | auction_1-2 | 645.6 μs | 642.9 μs | -0.4% | | auction_1-3 | 643.3 μs | 639.5 μs | -0.6% | | auction_1-4 | 235.5 μs | 233.0 μs | -1.1% | | auction_2-1 | 183.0 μs | 182.3 μs | -0.4% | | auction_2-2 | 645.0 μs | 642.6 μs | -0.4% | | auction_2-3 | 844.5 μs | 838.6 μs | -0.7% | | auction_2-4 | 644.8 μs | 633.0 μs | -1.8% | | auction_2-5 | 234.6 μs | 233.2 μs | -0.6% | | crowdfunding-success-1 | 214.0 μs | 212.2 μs | -0.8% | | crowdfunding-success-2 | 214.2 μs | 212.2 μs | -0.9% | | crowdfunding-success-3 | 214.4 μs | 212.3 μs | -1.0% | | currency-1 | 250.7 μs | 248.6 μs | -0.8% | | escrow-redeem_1-1 | 354.2 μs | 350.9 μs | -0.9% | | escrow-redeem_1-2 | 354.4 μs | 351.7 μs | -0.8% | | escrow-redeem_2-1 | 411.4 μs | 404.5 μs | -1.7% | | escrow-redeem_2-2 | 413.3 μs | 409.1 μs | -1.0% | | escrow-redeem_2-3 | 412.8 μs | 413.8 μs | +0.2% | | escrow-refund-1 | 158.1 μs | 158.7 μs | +0.4% | | future-increase-margin-1 | 251.7 μs | 251.7 μs | 0.0% | | future-increase-margin-2 | 541.0 μs | 544.2 μs | +0.6% | | future-increase-margin-3 | 542.0 μs | 546.1 μs | +0.8% | | future-increase-margin-4 | 491.5 μs | 496.6 μs | +1.0% | | future-increase-margin-5 | 820.9 μs | 826.3 μs | +0.7% | | future-pay-out-1 | 249.1 μs | 251.2 μs | +0.8% | | future-pay-out-2 | 541.2 μs | 545.9 μs | +0.9% | | future-pay-out-3 | 541.3 μs | 545.1 μs | +0.7% | | future-pay-out-4 | 819.9 μs | 822.8 μs | +0.4% | | future-settle-early-1 | 249.2 μs | 251.2 μs | +0.8% | | future-settle-early-2 | 539.8 μs | 543.0 μs | +0.6% | | future-settle-early-3 | 540.6 μs | 545.3 μs | +0.9% | | future-settle-early-4 | 614.1 μs | 604.0 μs | -1.6% | | game-sm-success_1-1 | 392.1 μs | 386.6 μs | -1.4% | | game-sm-success_1-2 | 202.1 μs | 198.6 μs | -1.7% | | game-sm-success_1-3 | 642.9 μs | 633.3 μs | -1.5% | | game-sm-success_1-4 | 229.7 μs | 227.0 μs | -1.2% | | game-sm-success_2-1 | 392.9 μs | 386.9 μs | -1.5% | | game-sm-success_2-2 | 202.2 μs | 198.8 μs | -1.7% | | game-sm-success_2-3 | 643.2 μs | 633.7 μs | -1.5% | | game-sm-success_2-4 | 229.9 μs | 227.6 μs | -1.0% | | game-sm-success_2-5 | 642.2 μs | 644.2 μs | +0.3% | | game-sm-success_2-6 | 230.2 μs | 231.7 μs | +0.7% | | multisig-sm-1 | 400.7 μs | 404.5 μs | +0.9% | | multisig-sm-2 | 388.7 μs | 392.6 μs | +1.0% | | multisig-sm-3 | 394.2 μs | 397.5 μs | +0.8% | | multisig-sm-4 | 397.2 μs | 401.7 μs | +1.1% | | multisig-sm-5 | 562.0 μs | 568.2 μs | +1.1% | | multisig-sm-6 | 401.5 μs | 403.8 μs | +0.6% | | multisig-sm-7 | 389.3 μs | 392.8 μs | +0.9% | | multisig-sm-8 | 393.7 μs | 397.5 μs | +1.0% | | multisig-sm-9 | 397.7 μs | 401.7 μs | +1.0% | | multisig-sm-10 | 562.5 μs | 568.9 μs | +1.1% | | ping-pong-1 | 329.9 μs | 333.3 μs | +1.0% | | ping-pong-2 | 330.1 μs | 333.6 μs | +1.1% | | ping-pong_2-1 | 202.0 μs | 203.8 μs | +0.9% | | prism-1 | 171.3 μs | 173.2 μs | +1.1% | | prism-2 | 413.8 μs | 414.9 μs | +0.3% | | prism-3 | 369.7 μs | 372.6 μs | +0.8% | | pubkey-1 | 144.0 μs | 144.8 μs | +0.6% | | stablecoin_1-1 | 914.1 μs | 918.8 μs | +0.5% | | stablecoin_1-2 | 198.0 μs | 199.4 μs | +0.7% | | stablecoin_1-3 | 1.052 ms | 1.055 ms | +0.3% | | stablecoin_1-4 | 210.3 μs | 211.0 μs | +0.3% | | stablecoin_1-5 | 1.350 ms | 1.357 ms | +0.5% | | stablecoin_1-6 | 258.8 μs | 261.5 μs | +1.0% | | stablecoin_2-1 | 912.9 μs | 919.4 μs | +0.7% | | stablecoin_2-2 | 197.3 μs | 199.8 μs | +1.3% | | stablecoin_2-3 | 1.049 ms | 1.054 ms | +0.5% | | stablecoin_2-4 | 209.8 μs | 211.3 μs | +0.7% | | token-account-1 | 191.3 μs | 192.8 μs | +0.8% | | token-account-2 | 338.2 μs | 341.3 μs | +0.9% | | uniswap-1 | 412.9 μs | 415.8 μs | +0.7% | | uniswap-2 | 222.8 μs | 224.4 μs | +0.7% | | uniswap-3 | 1.765 ms | 1.781 ms | +0.9% | | uniswap-4 | 338.2 μs | 341.0 μs | +0.8% | | uniswap-5 | 1.166 ms | 1.173 ms | +0.6% | | uniswap-6 | 325.5 μs | 327.2 μs | +0.5% | | vesting-1 | 351.3 μs | 352.4 μs | +0.3% | | | | | | | TOTAL | 35.52 ms | 35.58 ms | +0.2% |

[kwxm]

/benchmark nofib

[kwxm]

/benchmark nofib

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | clausify/formula1 | 3.064 ms | 3.148 ms | +2.7% | | clausify/formula2 | 4.060 ms | 4.218 ms | +3.9% | | clausify/formula3 | 11.14 ms | 11.58 ms | +3.9% | | clausify/formula4 | 25.67 ms | 26.81 ms | +4.4% | | clausify/formula5 | 55.38 ms | 56.49 ms | +2.0% | | knights/4x4 | 17.90 ms | 18.85 ms | +5.3% | | knights/6x6 | 46.56 ms | 48.49 ms | +4.1% | | knights/8x8 | 81.68 ms | 85.29 ms | +4.4% | | primetest/05digits | 10.30 ms | 10.82 ms | +5.0% | | primetest/10digits | 20.12 ms | 21.07 ms | +4.7% | | primetest/30digits | 60.53 ms | 64.75 ms | +7.0% | | primetest/50digits | 102.7 ms | 107.7 ms | +4.9% | | queens4x4/bt | 5.403 ms | 5.577 ms | +3.2% | | queens4x4/bm | 6.886 ms | 7.106 ms | +3.2% | | queens4x4/bjbt1 | 6.575 ms | 6.779 ms | +3.1% | | queens4x4/bjbt2 | 6.212 ms | 6.340 ms | +2.1% | | queens4x4/fc | 13.89 ms | 14.46 ms | +4.1% | | queens5x5/bt | 73.46 ms | 76.10 ms | +3.6% | | queens5x5/bm | 77.07 ms | 79.43 ms | +3.1% | | queens5x5/bjbt1 | 85.61 ms | 88.73 ms | +3.6% | | queens5x5/bjbt2 | 83.06 ms | 86.17 ms | +3.7% | | queens5x5/fc | 175.8 ms | 182.7 ms | +3.9% | | | | | | | TOTAL | 973.1 ms | 1.013 s | +4.1% |

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | clausify/formula1 | 3.111 ms | 3.113 ms | +0.1% | | clausify/formula2 | 4.149 ms | 4.167 ms | +0.4% | | clausify/formula3 | 11.38 ms | 11.44 ms | +0.5% | | clausify/formula4 | 25.84 ms | 25.98 ms | +0.5% | | clausify/formula5 | 54.96 ms | 55.40 ms | +0.8% | | knights/4x4 | 17.84 ms | 17.89 ms | +0.3% | | knights/6x6 | 46.46 ms | 46.70 ms | +0.5% | | knights/8x8 | 81.41 ms | 81.60 ms | +0.2% | | primetest/05digits | 10.20 ms | 10.26 ms | +0.6% | | primetest/10digits | 19.88 ms | 20.01 ms | +0.7% | | primetest/30digits | 61.30 ms | 61.64 ms | +0.6% | | primetest/50digits | 101.8 ms | 102.8 ms | +1.0% | | queens4x4/bt | 5.367 ms | 5.395 ms | +0.5% | | queens4x4/bm | 6.838 ms | 6.881 ms | +0.6% | | queens4x4/bjbt1 | 6.536 ms | 6.566 ms | +0.5% | | queens4x4/bjbt2 | 6.112 ms | 6.146 ms | +0.6% | | queens4x4/fc | 13.81 ms | 14.05 ms | +1.7% | | queens5x5/bt | 72.91 ms | 73.28 ms | +0.5% | | queens5x5/bm | 76.61 ms | 76.95 ms | +0.4% | | queens5x5/bjbt1 | 85.22 ms | 85.52 ms | +0.4% | | queens5x5/bjbt2 | 82.66 ms | 83.02 ms | +0.4% | | queens5x5/fc | 175.1 ms | 175.9 ms | +0.5% | | | | | | | TOTAL | 969.5 ms | 974.7 ms | +0.5% |

[kwxm]

/benchmark nofib

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | clausify/formula1 | 3.082 ms | 3.033 ms | -1.6% | | clausify/formula2 | 4.117 ms | 4.060 ms | -1.4% | | clausify/formula3 | 11.27 ms | 11.26 ms | -0.1% | | clausify/formula4 | 25.64 ms | 25.62 ms | -0.1% | | clausify/formula5 | 54.65 ms | 54.44 ms | -0.4% | | knights/4x4 | 17.72 ms | 17.63 ms | -0.5% | | knights/6x6 | 46.12 ms | 46.00 ms | -0.3% | | knights/8x8 | 79.55 ms | 80.61 ms | +1.3% | | primetest/05digits | 10.08 ms | 10.07 ms | -0.1% | | primetest/10digits | 19.78 ms | 19.64 ms | -0.7% | | primetest/30digits | 61.54 ms | 60.48 ms | -1.7% | | primetest/50digits | 102.4 ms | 100.6 ms | -1.8% | | queens4x4/bt | 5.341 ms | 5.306 ms | -0.7% | | queens4x4/bm | 6.883 ms | 6.771 ms | -1.6% | | queens4x4/bjbt1 | 6.501 ms | 6.483 ms | -0.3% | | queens4x4/bjbt2 | 6.081 ms | 6.051 ms | -0.5% | | queens4x4/fc | 13.72 ms | 13.69 ms | -0.2% | | queens5x5/bt | 72.62 ms | 72.18 ms | -0.6% | | queens5x5/bm | 76.18 ms | 75.83 ms | -0.5% | | queens5x5/bjbt1 | 84.79 ms | 84.45 ms | -0.4% | | queens5x5/bjbt2 | 82.15 ms | 81.98 ms | -0.2% | | queens5x5/fc | 173.7 ms | 173.7 ms | 0.0% | | | | | | | TOTAL | 963.9 ms | 959.9 ms | -0.4% |

[kwxm]

/benchmark nofib

[kwxm]

/benchmark validation

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | clausify/formula1 | 3.016 ms | 2.998 ms | -0.6% | | clausify/formula2 | 4.048 ms | 4.023 ms | -0.6% | | clausify/formula3 | 11.06 ms | 11.06 ms | 0.0% | | clausify/formula4 | 25.19 ms | 25.22 ms | +0.1% | | clausify/formula5 | 53.60 ms | 53.51 ms | -0.2% | | knights/4x4 | 17.31 ms | 17.32 ms | +0.1% | | knights/6x6 | 45.46 ms | 45.35 ms | -0.2% | | knights/8x8 | 80.22 ms | 79.47 ms | -0.9% | | primetest/05digits | 10.15 ms | 10.02 ms | -1.3% | | primetest/10digits | 19.81 ms | 19.63 ms | -0.9% | | primetest/30digits | 61.15 ms | 60.52 ms | -1.0% | | primetest/50digits | 101.3 ms | 100.6 ms | -0.7% | | queens4x4/bt | 5.367 ms | 5.336 ms | -0.6% | | queens4x4/bm | 6.681 ms | 6.766 ms | +1.3% | | queens4x4/bjbt1 | 6.399 ms | 6.458 ms | +0.9% | | queens4x4/bjbt2 | 5.989 ms | 6.053 ms | +1.1% | | queens4x4/fc | 13.54 ms | 13.64 ms | +0.7% | | queens5x5/bt | 72.26 ms | 72.21 ms | -0.1% | | queens5x5/bm | 75.11 ms | 75.92 ms | +1.1% | | queens5x5/bjbt1 | 83.72 ms | 84.66 ms | +1.1% | | queens5x5/bjbt2 | 80.75 ms | 82.29 ms | +1.9% | | queens5x5/fc | 171.0 ms | 174.1 ms | +1.8% | | | | | | | TOTAL | 953.1 ms | 957.2 ms | +0.4% |

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Comparing benchmark results of 'validation' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | auction_1-1 | 178.7 μs | 179.0 μs | +0.2% | | auction_1-2 | 631.3 μs | 632.1 μs | +0.1% | | auction_1-3 | 628.6 μs | 627.9 μs | -0.1% | | auction_1-4 | 230.0 μs | 230.6 μs | +0.3% | | auction_2-1 | 178.7 μs | 180.2 μs | +0.8% | | auction_2-2 | 631.0 μs | 632.0 μs | +0.2% | | auction_2-3 | 823.5 μs | 824.5 μs | +0.1% | | auction_2-4 | 627.6 μs | 626.5 μs | -0.2% | | auction_2-5 | 229.6 μs | 230.1 μs | +0.2% | | crowdfunding-success-1 | 209.1 μs | 209.5 μs | +0.2% | | crowdfunding-success-2 | 209.1 μs | 209.3 μs | +0.1% | | crowdfunding-success-3 | 209.6 μs | 210.8 μs | +0.6% | | currency-1 | 244.6 μs | 246.4 μs | +0.7% | | escrow-redeem_1-1 | 345.4 μs | 347.3 μs | +0.6% | | escrow-redeem_1-2 | 346.1 μs | 347.6 μs | +0.4% | | escrow-redeem_2-1 | 403.4 μs | 409.0 μs | +1.4% | | escrow-redeem_2-2 | 404.3 μs | 411.6 μs | +1.8% | | escrow-redeem_2-3 | 407.7 μs | 411.0 μs | +0.8% | | escrow-refund-1 | 156.2 μs | 157.3 μs | +0.7% | | future-increase-margin-1 | 248.6 μs | 249.6 μs | +0.4% | | future-increase-margin-2 | 538.6 μs | 539.7 μs | +0.2% | | future-increase-margin-3 | 540.1 μs | 540.0 μs | -0.0% | | future-increase-margin-4 | 492.3 μs | 493.0 μs | +0.1% | | future-increase-margin-5 | 819.5 μs | 821.4 μs | +0.2% | | future-pay-out-1 | 248.6 μs | 249.4 μs | +0.3% | | future-pay-out-2 | 537.9 μs | 537.7 μs | -0.0% | | future-pay-out-3 | 541.4 μs | 539.3 μs | -0.4% | | future-pay-out-4 | 818.4 μs | 818.0 μs | -0.0% | | future-settle-early-1 | 249.0 μs | 248.7 μs | -0.1% | | future-settle-early-2 | 536.9 μs | 537.5 μs | +0.1% | | future-settle-early-3 | 538.2 μs | 539.5 μs | +0.2% | | future-settle-early-4 | 610.6 μs | 608.2 μs | -0.4% | | game-sm-success_1-1 | 390.2 μs | 390.0 μs | -0.1% | | game-sm-success_1-2 | 201.1 μs | 201.7 μs | +0.3% | | game-sm-success_1-3 | 641.1 μs | 640.8 μs | -0.0% | | game-sm-success_1-4 | 229.2 μs | 229.6 μs | +0.2% | | game-sm-success_2-1 | 389.7 μs | 390.4 μs | +0.2% | | game-sm-success_2-2 | 201.2 μs | 201.6 μs | +0.2% | | game-sm-success_2-3 | 640.5 μs | 639.6 μs | -0.1% | | game-sm-success_2-4 | 228.9 μs | 229.4 μs | +0.2% | | game-sm-success_2-5 | 641.0 μs | 641.4 μs | +0.1% | | game-sm-success_2-6 | 229.6 μs | 230.0 μs | +0.2% | | multisig-sm-1 | 400.4 μs | 399.2 μs | -0.3% | | multisig-sm-2 | 389.2 μs | 384.8 μs | -1.1% | | multisig-sm-3 | 391.8 μs | 388.6 μs | -0.8% | | multisig-sm-4 | 395.4 μs | 390.6 μs | -1.2% | | multisig-sm-5 | 559.9 μs | 550.8 μs | -1.6% | | multisig-sm-6 | 399.6 μs | 395.2 μs | -1.1% | | multisig-sm-7 | 388.1 μs | 382.6 μs | -1.4% | | multisig-sm-8 | 392.5 μs | 386.7 μs | -1.5% | | multisig-sm-9 | 398.1 μs | 392.0 μs | -1.5% | | multisig-sm-10 | 559.4 μs | 553.2 μs | -1.1% | | ping-pong-1 | 330.4 μs | 325.2 μs | -1.6% | | ping-pong-2 | 327.9 μs | 324.9 μs | -0.9% | | ping-pong_2-1 | 201.5 μs | 198.1 μs | -1.7% | | prism-1 | 170.4 μs | 168.0 μs | -1.4% | | prism-2 | 410.1 μs | 404.2 μs | -1.4% | | prism-3 | 367.6 μs | 363.1 μs | -1.2% | | pubkey-1 | 142.9 μs | 141.5 μs | -1.0% | | stablecoin_1-1 | 909.0 μs | 895.3 μs | -1.5% | | stablecoin_1-2 | 196.5 μs | 194.7 μs | -0.9% | | stablecoin_1-3 | 1.043 ms | 1.029 ms | -1.3% | | stablecoin_1-4 | 208.4 μs | 206.2 μs | -1.1% | | stablecoin_1-5 | 1.340 ms | 1.325 ms | -1.1% | | stablecoin_1-6 | 258.1 μs | 255.3 μs | -1.1% | | stablecoin_2-1 | 907.1 μs | 896.5 μs | -1.2% | | stablecoin_2-2 | 197.0 μs | 194.8 μs | -1.1% | | stablecoin_2-3 | 1.041 ms | 1.031 ms | -1.0% | | stablecoin_2-4 | 208.0 μs | 206.0 μs | -1.0% | | token-account-1 | 190.7 μs | 188.2 μs | -1.3% | | token-account-2 | 336.7 μs | 331.9 μs | -1.4% | | uniswap-1 | 411.1 μs | 405.0 μs | -1.5% | | uniswap-2 | 221.2 μs | 219.4 μs | -0.8% | | uniswap-3 | 1.764 ms | 1.737 ms | -1.5% | | uniswap-4 | 336.1 μs | 333.4 μs | -0.8% | | uniswap-5 | 1.163 ms | 1.145 ms | -1.5% | | uniswap-6 | 323.2 μs | 319.3 μs | -1.2% | | vesting-1 | 350.9 μs | 344.4 μs | -1.9% | | | | | | | TOTAL | 35.24 ms | 35.05 ms | -0.5% |

[kwxm]

/benchmark nofib

[kwxm]

/benchmark validation

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Comparing benchmark results of 'nofib' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | clausify/formula1 | 3.110 ms | 3.040 ms | -2.3% | | clausify/formula2 | 4.166 ms | 4.100 ms | -1.6% | | clausify/formula3 | 11.41 ms | 11.15 ms | -2.3% | | clausify/formula4 | 26.00 ms | 25.44 ms | -2.2% | | clausify/formula5 | 55.47 ms | 54.19 ms | -2.3% | | knights/4x4 | 17.89 ms | 17.64 ms | -1.4% | | knights/6x6 | 46.82 ms | 45.53 ms | -2.8% | | knights/8x8 | 82.00 ms | 79.90 ms | -2.6% | | primetest/05digits | 10.31 ms | 10.08 ms | -2.2% | | primetest/10digits | 20.19 ms | 19.72 ms | -2.3% | | primetest/30digits | 62.01 ms | 60.80 ms | -2.0% | | primetest/50digits | 103.1 ms | 101.2 ms | -1.8% | | queens4x4/bt | 5.404 ms | 5.291 ms | -2.1% | | queens4x4/bm | 6.890 ms | 6.763 ms | -1.8% | | queens4x4/bjbt1 | 6.582 ms | 6.452 ms | -2.0% | | queens4x4/bjbt2 | 6.150 ms | 6.042 ms | -1.8% | | queens4x4/fc | 13.91 ms | 13.67 ms | -1.7% | | queens5x5/bt | 73.45 ms | 72.28 ms | -1.6% | | queens5x5/bm | 77.15 ms | 75.99 ms | -1.5% | | queens5x5/bjbt1 | 85.80 ms | 84.68 ms | -1.3% | | queens5x5/bjbt2 | 83.34 ms | 82.17 ms | -1.4% | | queens5x5/fc | 176.3 ms | 174.0 ms | -1.3% | | | | | | | TOTAL | 977.5 ms | 960.1 ms | -1.8% |

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Comparing benchmark results of 'validation' on 'ffe41b864' (base) and 'a6d976575' (PR)

Results table

| Script | ffe41b8 | a6d9765 | Change | | :------ | :------: | :------: | :------: | | auction_1-1 | 181.4 μs | 180.3 μs | -0.6% | | auction_1-2 | 638.6 μs | 632.9 μs | -0.9% | | auction_1-3 | 642.9 μs | 633.2 μs | -1.5% | | auction_1-4 | 236.3 μs | 231.9 μs | -1.9% | | auction_2-1 | 183.8 μs | 180.3 μs | -1.9% | | auction_2-2 | 648.3 μs | 637.2 μs | -1.7% | | auction_2-3 | 846.4 μs | 832.8 μs | -1.6% | | auction_2-4 | 644.0 μs | 632.5 μs | -1.8% | | auction_2-5 | 235.7 μs | 231.5 μs | -1.8% | | crowdfunding-success-1 | 214.3 μs | 210.7 μs | -1.7% | | crowdfunding-success-2 | 214.9 μs | 211.2 μs | -1.7% | | crowdfunding-success-3 | 214.9 μs | 211.2 μs | -1.7% | | currency-1 | 251.1 μs | 246.8 μs | -1.7% | | escrow-redeem_1-1 | 354.0 μs | 348.7 μs | -1.5% | | escrow-redeem_1-2 | 354.9 μs | 347.4 μs | -2.1% | | escrow-redeem_2-1 | 414.6 μs | 408.9 μs | -1.4% | | escrow-redeem_2-2 | 415.1 μs | 409.2 μs | -1.4% | | escrow-redeem_2-3 | 414.9 μs | 409.2 μs | -1.4% | | escrow-refund-1 | 158.1 μs | 156.3 μs | -1.1% | | future-increase-margin-1 | 251.9 μs | 248.6 μs | -1.3% | | future-increase-margin-2 | 546.4 μs | 538.8 μs | -1.4% | | future-increase-margin-3 | 545.6 μs | 539.0 μs | -1.2% | | future-increase-margin-4 | 498.4 μs | 490.7 μs | -1.5% | | future-increase-margin-5 | 826.7 μs | 815.5 μs | -1.4% | | future-pay-out-1 | 252.0 μs | 248.6 μs | -1.3% | | future-pay-out-2 | 545.9 μs | 537.1 μs | -1.6% | | future-pay-out-3 | 547.1 μs | 533.9 μs | -2.4% | | future-pay-out-4 | 826.9 μs | 808.8 μs | -2.2% | | future-settle-early-1 | 251.2 μs | 249.0 μs | -0.9% | | future-settle-early-2 | 544.8 μs | 539.6 μs | -1.0% | | future-settle-early-3 | 545.5 μs | 536.0 μs | -1.7% | | future-settle-early-4 | 617.8 μs | 607.9 μs | -1.6% | | game-sm-success_1-1 | 395.8 μs | 391.7 μs | -1.0% | | game-sm-success_1-2 | 203.8 μs | 201.6 μs | -1.1% | | game-sm-success_1-3 | 649.2 μs | 640.5 μs | -1.3% | | game-sm-success_1-4 | 231.1 μs | 228.8 μs | -1.0% | | game-sm-success_2-1 | 393.7 μs | 388.3 μs | -1.4% | | game-sm-success_2-2 | 203.2 μs | 201.0 μs | -1.1% | | game-sm-success_2-3 | 645.0 μs | 635.3 μs | -1.5% | | game-sm-success_2-4 | 231.3 μs | 227.0 μs | -1.9% | | game-sm-success_2-5 | 646.5 μs | 637.7 μs | -1.4% | | game-sm-success_2-6 | 230.8 μs | 230.2 μs | -0.3% | | multisig-sm-1 | 403.4 μs | 397.5 μs | -1.5% | | multisig-sm-2 | 391.2 μs | 384.9 μs | -1.6% | | multisig-sm-3 | 397.2 μs | 390.5 μs | -1.7% | | multisig-sm-4 | 400.7 μs | 393.7 μs | -1.7% | | multisig-sm-5 | 565.9 μs | 559.8 μs | -1.1% | | multisig-sm-6 | 403.6 μs | 397.5 μs | -1.5% | | multisig-sm-7 | 391.8 μs | 388.1 μs | -0.9% | | multisig-sm-8 | 395.8 μs | 389.3 μs | -1.6% | | multisig-sm-9 | 399.1 μs | 397.4 μs | -0.4% | | multisig-sm-10 | 565.1 μs | 556.3 μs | -1.6% | | ping-pong-1 | 331.7 μs | 327.8 μs | -1.2% | | ping-pong-2 | 328.1 μs | 328.2 μs | +0.0% | | ping-pong_2-1 | 199.5 μs | 201.1 μs | +0.8% | | prism-1 | 169.5 μs | 171.1 μs | +0.9% | | prism-2 | 407.1 μs | 412.2 μs | +1.3% | | prism-3 | 365.2 μs | 368.0 μs | +0.8% | | pubkey-1 | 142.1 μs | 144.0 μs | +1.3% | | stablecoin_1-1 | 901.4 μs | 905.9 μs | +0.5% | | stablecoin_1-2 | 195.7 μs | 199.1 μs | +1.7% | | stablecoin_1-3 | 1.036 ms | 1.058 ms | +2.1% | | stablecoin_1-4 | 206.9 μs | 211.9 μs | +2.4% | | stablecoin_1-5 | 1.330 ms | 1.357 ms | +2.0% | | stablecoin_1-6 | 256.5 μs | 262.6 μs | +2.4% | | stablecoin_2-1 | 907.9 μs | 919.8 μs | +1.3% | | stablecoin_2-2 | 196.9 μs | 199.7 μs | +1.4% | | stablecoin_2-3 | 1.044 ms | 1.054 ms | +1.0% | | stablecoin_2-4 | 208.5 μs | 211.6 μs | +1.5% | | token-account-1 | 190.7 μs | 193.0 μs | +1.2% | | token-account-2 | 336.8 μs | 340.9 μs | +1.2% | | uniswap-1 | 414.3 μs | 417.5 μs | +0.8% | | uniswap-2 | 224.6 μs | 223.9 μs | -0.3% | | uniswap-3 | 1.783 ms | 1.780 ms | -0.2% | | uniswap-4 | 340.6 μs | 341.4 μs | +0.2% | | uniswap-5 | 1.175 ms | 1.176 ms | +0.1% | | uniswap-6 | 328.3 μs | 328.6 μs | +0.1% | | vesting-1 | 355.5 μs | 354.5 μs | -0.3% | | | | | | | TOTAL | 35.58 ms | 35.37 ms | -0.6% |

[kwxm]

/benchmark nofib

[kwxm]

/benchmark validation

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Click here to check the status of your benchmark.

[kwxm]

/benchmark validation

[kwxm]

/benchmark nofib

[github-actions[bot]]

Click here to check the status of your benchmark.

[github-actions[bot]]

Click here to check the status of your benchmark.

[kwxm]

This is inconclusive. There's a lot of variation in the results but it's not clear if that's due to the reduction in benchmarking times or inconsistency in the machine.

[kwxm]

Also the benchmarks have stopped working because the base commit has vanished, so I'll close this and try again some other time.