Run bench on demo

[ffakenz]

This PR enhances the demo tutorial by enabling hydra-cluster benchmarks to run on an active Hydra cluster.

usage

nix run .#legacyPackages.x86_64-linux.hydra-cluster.components.benchmarks.bench-e2e -- \
          demo \
          --output-directory=$(pwd)/benchmarks \
          --scaling-factor=100 \
          --timeout=1000s \
          --testnet-magic 42 \
          --node-socket=${NETWORK_DIR}/node.socket \
          --hydra-client=localhost:4001 \
          --hydra-client=localhost:4002 \
          --hydra-client=localhost:4003

prerequisites

• A Cardano node must be running on specified node-socket.
• Hydra nodes must be operational on provided hydra-client hosts.
• There’s no need to pre-seed the keys, as the bench-demo script will automatically fund them using the faucet.
• Note that the reference scripts should already be published, and the Hydra nodes must be running with those scripts.

Todo

• [x] Fix the FIXME about > 33
• [x] Remove duplicate seeding
• [x] Make sure the entire CI process doesn't fail when the pumba causes the network to fail
• [x] Make it so that if it fails the head is closed.
• [x] Quick little matrix to run a few different scenarios
• [x] Make the bench-e2e fail if it didn't submit all the txns ( ideally would also be able to see visually in the job list; but Github is missing a feature see also https://github.com/actions/runner/issues/2347 )
• [x] Get docker info via docker inspect instead of parsing yaml (!)
• [x] Make sure results.csv is written to the outputDirectory not the tmp directory
• [x] Upload the results as part of the artifacts
• [x] Write the summary out even when it failed

---

• [x] CHANGELOG updated or not needed
• [x] Documentation updated or not needed
• [x] Haddocks updated or not needed
• [x] No new TODOs introduced or explained herafter

[github-actions[bot]]

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2024-08-29 13:04:41.231504601 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	2fac819a1f4f14e29639d1414220d2a18b6abd6b8e444d88d0dda8ff	3799
νCommit	2043a9f1a685bcf491413a5f139ee42e335157c8c6bc8d9e4018669d	1743
νHead	bd9fad235c871fb7f837c767593018a84be3083ff80f9dab5f1c55f9	10194
μHead	c8038945816586c4d38926ee63bba67821eb863794220ebbd0bf79ee*	4607

• The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

  Init transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	5188	5.81	2.30	0.44
2	5389	7.18	2.84	0.47
3	5588	8.46	3.34	0.49
5	5993	11.32	4.48	0.54
10	6998	18.20	7.20	0.66
56	16250	81.53	32.25	1.76

Commit transaction costs

This uses ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	559	10.52	4.15	0.29
2	747	13.86	5.65	0.34
3	931	17.33	7.20	0.38
5	1304	24.65	10.44	0.48
10	2247	45.22	19.36	0.75
20	4121	95.99	40.76	1.40

CollectCom transaction costs

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	549	22.14	8.66	0.42
2	113	663	32.96	13.05	0.54
3	169	769	44.09	17.69	0.67
4	227	879	60.01	24.20	0.85
5	283	994	78.11	31.66	1.05
6	336	1100	91.73	37.56	1.21

Cost of Decrement Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	615	17.71	7.79	0.38
2	847	20.18	9.47	0.42
3	945	21.33	10.65	0.44
5	1153	22.61	12.56	0.47
10	1925	31.25	19.56	0.63
50	8087	98.91	74.98	1.85

Close transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	659	21.02	9.43	0.42
2	735	22.07	10.47	0.44
3	862	23.46	11.84	0.46
5	1197	26.78	15.00	0.53
10	2043	35.48	23.25	0.69
50	8051	99.98	84.29	1.92

Contest transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	670	27.16	11.67	0.48
2	763	28.48	12.83	0.51
3	1022	31.06	15.03	0.55
5	1265	34.49	17.99	0.61
10	2091	44.83	26.75	0.80
38	5990	95.82	70.29	1.69

Abort transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	5055	17.36	7.55	0.57
2	5220	29.19	12.87	0.71
3	5320	41.54	18.32	0.85
4	5543	59.66	26.52	1.07
5	5672	77.35	34.44	1.27
6	5813	98.64	44.02	1.52

FanOut transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
5	0	0	5022	7.75	3.28	0.46
5	1	57	5057	9.08	4.08	0.48
5	5	285	5192	13.69	6.96	0.54
5	10	570	5362	18.87	10.31	0.61
5	20	1139	5702	30.38	17.51	0.77
5	30	1706	6041	42.10	24.80	0.94
5	40	2279	6383	53.03	31.76	1.09
5	50	2847	6721	64.37	38.89	1.25
5	81	4615	7776	99.53	61.01	1.74

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2024-08-29 13:06:50.382260433 UTC

Baseline Scenario

Number of nodes	1
Number of txs	3000
Avg. Confirmation Time (ms)	4.194986589
P99	8.141198899999969ms
P95	5.328280549999999ms
P50	3.9952555ms
Number of Invalid txs	0

Three local nodes

Number of nodes	3
Number of txs	9000
Avg. Confirmation Time (ms)	23.726739321
P99	104.91504140000006ms
P95	32.825234349999995ms
P50	21.1135175ms
Number of Invalid txs	0

[github-actions[bot]]

Test Results

469 tests  ±0   462 :white_check_mark: ±0   18m 21s :stopwatch: +58s 150 suites ±0     7 :zzz: ±0    5 files   ±0     0 :x: ±0 

Results for commit c5d12084. ± Comparison against base commit a51e04c8.

:recycle: This comment has been updated with latest results.

[noonio]

While Github doesn't show the specific matrix configuration fail, it does show it as an "annotation"; so that's extremely useful for verifiying that the scenarios we expect fail in the right way.

[noonio]

This is looking really great!