search

cardano-scaling / hydra

Implementation of the Hydra Head protocol
https://hydra.family/head-protocol/
Apache License 2.0
284 stars 86 forks source link

Small changes to run benchmarks for lots of transactions #1726

Closed noonio closed 1 week ago

noonio commented 4 weeks ago

Fixes #1724

This slightly modifies the micro benchmark to hake a configurable number of transactions to run for, and shows how to use it to get memory usage output.

github-actions[bot] commented 4 weeks ago

Transaction cost differences

No cost or size differences found

noonio commented 2 weeks ago

Hmm, seems like tasty-bench doesn't support the HTML output option; so will need to see if there's a quick fix for that.

github-actions[bot] commented 2 weeks ago

Test Results

  5 files  ±0  162 suites  ±0   28m 3s ⏱️ - 4m 7s 554 tests ±0  548 ✅ ±0  6 💤 ±0  0 ❌ ±0  556 runs  ±0  550 ✅ ±0  6 💤 ±0  0 ❌ ±0 

Results for commit 41fb994c. ± Comparison against base commit 3be7b22a.

:recycle: This comment has been updated with latest results.

noonio commented 2 weeks ago

Running the benchmarks like so:

N_TXNS=1000000 cabal bench micro --benchmark-options '--json output.json +RTS -T'

Supposing you get peakMbAllocated from this:

jq '.[2][0].reportKeys[5]' hydra-node/output.json

You can find the mb this way:

jq '.[2][0].reportMeasured[0][5]' hydra-node/output.json

(Exercise for someone to do this properly for the JSON output.)

github-actions[bot] commented 2 weeks ago

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at 2024-11-13 13:45:59.043412945 UTC
Max. memory units 14000000
Max. CPU units 10000000000
Max. tx size (kB) 16384

Script summary

Name Hash Size (Bytes)
νInitial 00a6ddbc130ab92f5b7cb8d1ccd8d79eca5bfe25f6843c07b62841f0 2667
νCommit 3e5a776bcee213e3dfd15806952a10ac5590e3e97d09d62eb99266b2 690
νHead 00e0e6aceb51fcb4fcc5dc6032cc524a641163f0723b25d995de3ae8 12633
μHead 134f18365d53de72b07af2e703c96d8a6f0643c71d4c1a2050b8182a* 11111
νDeposit 2feb47889a4f658dc593cefcb0e37d584b9431944f08a687f3dab4af 4865
Parties Tx size % max Mem % max CPU Min fee ₳
1 11700 8.83 2.88 0.76
2 11896 10.76 3.52 0.79
3 12097 12.83 4.21 0.82
5 12500 15.89 5.17 0.87
10 13503 24.65 8.02 1.00
24 16322 49.59 16.14 1.39

Commit transaction costs

This uses ada-only outputs for better comparability.

UTxO Tx size % max Mem % max CPU Min fee ₳
1 561 2.45 1.17 0.20
2 743 3.40 1.74 0.22
3 917 4.39 2.34 0.24
5 1274 6.46 3.61 0.28
10 2180 12.24 7.28 0.40
54 10063 99.20 68.72 1.89

CollectCom transaction costs

Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 57 560 24.69 7.19 0.43
2 114 671 32.74 9.53 0.51
3 170 786 40.68 11.83 0.60
4 227 893 51.48 14.81 0.71
5 283 1004 62.90 17.95 0.83
6 340 1116 63.48 18.49 0.85
7 395 1227 73.24 21.27 0.95
8 451 1338 85.09 24.42 1.07
9 505 1449 87.27 25.45 1.10

Cost of Decrement Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 654 22.26 7.22 0.41
2 771 23.99 8.36 0.44
3 914 24.67 9.20 0.46
5 1248 28.77 11.68 0.52
10 1911 36.87 17.31 0.66
41 6783 99.53 55.53 1.65

Close transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 668 24.31 8.01 0.43
2 819 26.44 9.44 0.47
3 1036 28.61 10.90 0.51
5 1286 31.60 13.19 0.56
10 1998 39.80 19.23 0.70
44 7062 99.33 61.92 1.71

Contest transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 688 30.32 9.61 0.49
2 815 32.17 10.85 0.52
3 900 33.42 11.80 0.54
5 1344 38.85 15.13 0.63
10 1927 47.73 21.14 0.77
34 5529 97.00 53.20 1.56

Abort transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties Tx size % max Mem % max CPU Min fee ₳
1 11575 25.80 8.76 0.93
2 11708 34.12 11.62 1.02
3 11861 43.69 14.91 1.13
4 12013 52.44 17.86 1.23
5 12035 56.15 19.08 1.27
6 12317 71.23 24.34 1.44
7 12431 79.38 27.06 1.53
8 12405 83.59 28.31 1.58
9 12525 87.13 29.54 1.62
10 12712 95.88 32.70 1.72

FanOut transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTXO for better comparability.

Parties UTxO UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
10 0 0 11692 17.34 5.93 0.85
10 1 57 11727 19.19 6.70 0.87
10 5 285 11862 27.49 10.07 0.97
10 10 570 12032 35.13 13.28 1.06
10 20 1139 12371 54.02 21.03 1.28
10 30 1704 12708 73.27 28.92 1.51
10 40 2273 13047 91.81 36.54 1.73
10 44 2505 13187 98.49 39.32 1.81

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2024-11-13 13:49:05.436458294 UTC

Baseline Scenario

Number of nodes 1
Number of txs 300
Avg. Confirmation Time (ms) 4.725672723
P99 7.645229199999999ms
P95 6.0821347999999995ms
P50 4.552252ms
Number of Invalid txs 0

Three local nodes

Number of nodes 3
Number of txs 900
Avg. Confirmation Time (ms) 24.767187944
P99 114.54338578999996ms
P95 34.76249535ms
P50 22.1581765ms
Number of Invalid txs 0