cardano-scaling / hydra

Implementation of the Hydra Head protocol
https://hydra.family/head-protocol/
Apache License 2.0
281 stars 86 forks source link

Golden tests for persisted StateEvent values #1554

Closed ch1bo closed 1 month ago

ch1bo commented 2 months ago

Why

We do not want to accidentally break the interface of persisted data and inform users up-front when there is a BREAKING change in the changelog.

What

Adding golden tests to test for potential regressions in the serialization of StateEvent and its constituents. This, plus being vigilant on changes to golden files, allows us to communicate or avoid any breaking changes.

How

By using the ADT variants of hspec-golden-aeson we can ensure all data constructors are covered. See also https://hackage.haskell.org/package/hspec-golden-aeson-0.9.0.0/docs/Test-Aeson-GenericSpecs.html#v:roundtripAndGoldenADTSpecs

This comes at the cost of more golden files, but decreasing sample size to 1 should make this somewhat bearable.

Bonus: this improves and tests some of our generators in hydra-cardano-api


github-actions[bot] commented 2 months ago

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at 2024-09-02 14:04:37.20987018 UTC
Max. memory units 14000000
Max. CPU units 10000000000
Max. tx size (kB) 16384

Script summary

Name Hash Size (Bytes)
νInitial 2fac819a1f4f14e29639d1414220d2a18b6abd6b8e444d88d0dda8ff 3799
νCommit 2043a9f1a685bcf491413a5f139ee42e335157c8c6bc8d9e4018669d 1743
νHead bd9fad235c871fb7f837c767593018a84be3083ff80f9dab5f1c55f9 10194
μHead c8038945816586c4d38926ee63bba67821eb863794220ebbd0bf79ee* 4607
Parties Tx size % max Mem % max CPU Min fee ₳
1 5189 5.81 2.30 0.44
2 5389 7.03 2.77 0.46
3 5590 8.56 3.39 0.49
5 5998 11.22 4.43 0.54
10 6999 18.47 7.31 0.66
56 16247 81.43 32.21 1.76

Commit transaction costs

This uses ada-only outputs for better comparability.

UTxO Tx size % max Mem % max CPU Min fee ₳
1 559 10.52 4.15 0.29
2 748 13.86 5.65 0.34
3 937 17.33 7.20 0.38
5 1309 24.65 10.44 0.48
10 2238 45.22 19.36 0.75
20 4116 95.99 40.76 1.40

CollectCom transaction costs

Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 57 549 22.14 8.66 0.42
2 114 663 32.96 13.05 0.54
3 170 769 46.33 18.52 0.69
4 226 879 61.38 24.72 0.86
5 283 989 72.63 29.64 0.99
6 340 1100 98.01 39.89 1.28

Cost of Decrement Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 618 17.93 7.87 0.38
2 753 19.11 9.07 0.40
3 916 20.43 10.32 0.43
5 1265 24.36 13.21 0.50
10 2016 31.93 19.81 0.64
50 7661 93.01 72.82 1.76

Close transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 671 20.98 9.42 0.42
2 774 22.41 10.78 0.44
3 970 24.26 12.52 0.48
5 1262 27.08 15.30 0.53
10 1947 34.85 22.64 0.68
49 8019 98.72 83.15 1.90

Contest transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 684 27.14 11.67 0.48
2 847 29.24 13.47 0.52
3 1004 31.05 15.02 0.55
5 1315 35.02 18.37 0.62
10 2246 45.61 27.38 0.81
38 6026 95.73 70.50 1.70

Abort transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties Tx size % max Mem % max CPU Min fee ₳
1 5059 17.43 7.58 0.57
2 5132 27.58 12.01 0.69
3 5363 42.82 18.95 0.87
4 5492 58.44 25.91 1.05
5 5639 77.42 34.46 1.27
6 5812 94.44 42.02 1.47

FanOut transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties UTxO UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
5 0 0 5022 7.36 3.11 0.45
5 1 57 5057 8.88 3.99 0.47
5 5 285 5192 13.60 6.92 0.54
5 10 569 5362 19.06 10.39 0.62
5 20 1139 5702 30.58 17.60 0.78
5 30 1706 6040 41.70 24.63 0.93
5 40 2275 6379 53.03 31.76 1.09
5 50 2848 6722 64.37 38.89 1.25
5 81 4611 7773 99.53 61.01 1.74

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2024-09-02 14:06:32.358552265 UTC

Baseline Scenario

Number of nodes 1
Number of txs 3000
Avg. Confirmation Time (ms) 4.640682190
P99 9.154680719999982ms
P95 6.580600049999997ms
P50 4.221868000000001ms
Number of Invalid txs 0

Three local nodes

Number of nodes 3
Number of txs 9000
Avg. Confirmation Time (ms) 24.000880903
P99 46.91106044000001ms
P95 32.40865785ms
P50 21.946295499999998ms
Number of Invalid txs 0
github-actions[bot] commented 2 months ago

Test Results

489 tests  +20   482 :white_check_mark: +20   16m 50s :stopwatch: - 1m 32s 152 suites + 2     7 :zzz: ± 0    5 files   ± 0     0 :x: ± 0 

Results for commit d1ab629e. ± Comparison against base commit b03fa32f.

This pull request removes 5 and adds 25 tests. Note that renamed tests count towards both. ``` Hydra.Chain.Direct.State/ChainState/JSON encoding of ChainState ‑ allows to encode values with aeson and read them back Hydra.Chain.Direct.State/ChainState/JSON encoding of ChainState ‑ produces the same JSON as is found in golden/ChainState.json Hydra.Chain.Direct.State/Plutus.PubKeyHash/JSON encoding of PubKeyHash ‑ allows to encode values with aeson and read them back Hydra.Chain.Direct.State/Plutus.PubKeyHash/JSON encoding of PubKeyHash ‑ produces the same JSON as is found in golden/PubKeyHash.json Hydra.Ledger.Cardano/PParamss ‑ Roundtrip JSON encoding ``` ``` Hydra.Chain.Direct.State/JSON encoding of PubKeyHash ‑ allows to encode values with aeson and read them back Hydra.Chain.Direct.State/JSON encoding of PubKeyHash ‑ produces the same JSON as is found in golden/PubKeyHash.json Hydra.Events.FileBased/persisted event format/JSON encoding of (StateEvent (Tx BabbageEra)) ‑ allows to encode values with aeson and read them back Hydra.Events.FileBased/persisted event format/JSON encoding of (StateEvent (Tx BabbageEra)) ‑ produces the same JSON as is found in golden/StateEvent (Tx BabbageEra).json Hydra.Events.FileBased/persisted event format/JSON encoding of StateChanged ‑ allows to encode values with aeson and read them back Hydra.Events.FileBased/persisted event format/JSON encoding of StateChanged ‑ produces the same JSON as is found in golden/StateChanged/ChainRolledBack.json Hydra.Events.FileBased/persisted event format/JSON encoding of StateChanged ‑ produces the same JSON as is found in golden/StateChanged/CommittedUTxO.json Hydra.Events.FileBased/persisted event format/JSON encoding of StateChanged ‑ produces the same JSON as is found in golden/StateChanged/DecommitFinalized.json Hydra.Events.FileBased/persisted event format/JSON encoding of StateChanged ‑ produces the same JSON as is found in golden/StateChanged/DecommitRecorded.json Hydra.Events.FileBased/persisted event format/JSON encoding of StateChanged ‑ produces the same JSON as is found in golden/StateChanged/HeadAborted.json … ```

:recycle: This comment has been updated with latest results.

ch1bo commented 2 months ago

@locallycompact You might be delighted to see more genXXX top-level generators :)