search

input-output-hk / hydra

Implementation of the Hydra Head protocol
https://hydra.family/head-protocol/
Apache License 2.0
264 stars 86 forks source link

Switch bench-e2e to JSON datasets again #1433

Closed ch1bo closed 1 month ago

ch1bo commented 1 month ago

This keeps the cardano-cli style transaction envelopes for transactions with essentially base16 encoded tx data, but uses JSON otherwise.

Having JSON encoded datasets allows easier introspection (title, description, number of txs) of individual parts of a dataset.

Note that this also fixes the broken titles in https://hydra.family/head-protocol/unstable/benchmarks/end-to-end-benchmarks/ as each dataset has a title.

ch1bo commented 1 month ago

@v0d1ch Created this to also ensure that https://github.com/input-output-hk/hydra/pull/1409 does not mess with our transaction decoding unexpectedly.

github-actions[bot] commented 1 month ago

Transactions Costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at 2024-05-13 15:59:58.716529766 UTC
Max. memory units 14000000
Max. CPU units 10000000000
Max. tx size (kB) 16384

Script summary

Name Hash Size (Bytes)
νInitial bccf2a430c016bc960fbf31b02694011cd399d20da8882aac9d33611 4110
νCommit 56b0f0b597150e619c76bed60683f3b1e42d7bc0685ed951b882bfc5 1975
νHead 86bff95ba20e9d1d1b34899a56d86bbacc9fed999260b27dcc92d128 9351
μHead 88f533cf67cd0fc93d7d9ccf0a8b1d69ffd1208a825efbebbc1d36ba* 4213
Parties Tx size % max Mem % max CPU Min fee ₳
1 4795 8.72 3.33 0.46
2 4996 10.91 4.18 0.49
3 5196 12.79 4.90 0.52
5 5600 16.72 6.41 0.58
10 6604 26.12 10.01 0.72
47 14043 98.21 37.70 1.83

Cost of Commit Transaction

This is using ada-only outputs for better comparability.

UTxO Tx size % max Mem % max CPU Min fee ₳
1 559 10.24 4.04 0.29
2 746 13.88 5.64 0.34
3 937 17.66 7.29 0.39
5 1313 25.66 10.74 0.49
10 2249 48.19 20.30 0.78
19 3923 97.83 40.79 1.41

Cost of CollectCom Transaction

Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 57 544 16.93 6.68 0.36
2 114 654 26.88 10.71 0.47
3 169 764 38.13 15.34 0.60
4 226 878 54.16 21.83 0.78
5 282 984 69.01 28.03 0.95
6 338 1095 82.56 33.83 1.11

Cost of Close Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 597 8.30 4.60 0.28
2 734 8.91 5.62 0.30
3 890 9.75 6.75 0.32
5 1215 11.35 9.14 0.36
10 2026 16.80 15.87 0.49
50 7841 50.05 62.59 1.35

Cost of Contest Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 577 8.24 4.39 0.27
2 746 9.34 5.78 0.30
3 959 10.48 7.21 0.33
5 1251 12.42 9.69 0.38
10 2054 17.37 16.10 0.50
50 8233 53.18 64.66 1.41

Cost of Abort Transaction

Some variation because of random mixture of still initial and already committed outputs.

Parties Tx size % max Mem % max CPU Min fee ₳
1 4666 17.09 7.39 0.55
2 4782 27.79 12.11 0.67
3 4917 40.55 17.77 0.82
4 5027 56.51 24.77 1.01
5 5100 67.27 29.30 1.13
6 5293 95.74 42.14 1.46

Cost of FanOut Transaction

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties UTxO UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
5 0 0 4627 8.06 3.37 0.44
5 1 57 4661 9.00 3.99 0.46
5 5 285 4798 13.82 6.93 0.52
5 10 568 4965 18.95 10.22 0.60
5 20 1138 5306 30.69 17.44 0.76
5 30 1704 5644 42.01 24.49 0.91
5 40 2277 5987 53.96 31.81 1.08
5 50 2845 6325 65.29 38.86 1.24
5 79 4498 7310 99.00 59.68 1.70

End-To-End Benchmark Results

This page is intended to collect the latest end-to-end benchmarks results produced by Hydra's Continuous Integration system from the latest master code.

Please take those results with a grain of salt as they are currently produced from very limited cloud VMs and not controlled hardware. Instead of focusing on the absolute results, the emphasis should be on relative results, eg. how the timings for a scenario evolve as the code changes.

Generated at 2024-05-13 16:02:50.021147497 UTC

Baseline Scenario

Number of nodes 1
Number of txs 3000
Avg. Confirmation Time (ms) 3.636125948
P99 5.497634109999991ms
P95 4.28879875ms
P50 3.513503ms
Number of Invalid txs 0

Three local nodes

Number of nodes 3
Number of txs 9000
Avg. Confirmation Time (ms) 20.895142725
P99 63.165024110000125ms
P95 28.330992649999995ms
P50 18.568966000000003ms
Number of Invalid txs 0
github-actions[bot] commented 1 month ago

Test Results

427 tests  +1   419 :white_check_mark: +1   15m 15s :stopwatch: -21s 139 suites +1     8 :zzz: ±0    2 files   ±0     0 :x: ±0 

Results for commit 101b7705. ± Comparison against base commit d8d2f93a.