input-output-hk / marlowe

Prototype implementation of domain-specific language for the design of smart-contracts over cryptocurrencies
Apache License 2.0
172 stars 43 forks source link

SCP-4860 - Make tests reproducible #158

Closed hrajchert closed 1 year ago

hrajchert commented 1 year ago

In order to make the tests reproducible the driver now includes a seed and size parameter to the GenerateRandomValue request. If a client process returns the same value for the same parameters, then you can reproduce a failure with --quickcheck-replay SEED.

In order to test this PR, It is recommended to also check out the corresponding purescript-marlowe PR, then:

Make sure the local tests are reproducible

  1. Modify a ToJSON definition in MarloweCoreJson.lhs to make sure the test fails
  2. Run the tests with cabal test marlowe-spec-test
  3. Run the tests again and see that the random value has changed
  4. Use the pattern and seed printed by the test and see check that the result is the same. To pass parameters to the local test you need to use --test-options like this:
cabal test marlowe-spec-test --test-options="--pattern /TransactionOutput/ --quickcheck-replay=457567"

Make sure client process tests are reproducible

  1. In the purescript-marlowe repository, modify the flake.nix input to use this version of marlowe-spec
    marloweSpec = {
      type = "github";
      owner = "input-output-hk";
      repo = "marlowe";
      ref = "hrajchert/scp-4860-make-tests-reproducible";
    };
  2. Enter nix develop in the purescript-marlowe repo
  3. Execute marlowe-test to build the project and run the tests (they should all pass)
  4. Modify a DecodeJson inside the Language.Marlowe.Core.V1.Semantics.Types module
  5. Execute marlowe-test, one of the serialization test should fail
  6. Execute cat $(which marlowe-test) and execute the last line plus the suggested pattern and replay seed
    # Something like
    $ /nix/store/5zvw46q48pl978lmkciygdc5f258a5m3-marlowe-spec-test-exe-marlowe-spec-0.1.0.0/bin/marlowe-spec -c /nix/store/qbqpb20y3z8k3167hy7slws0ic5wc6ph-marlowe-spec-client/bin/marlowe-spec-client -p '/TransactionOutput/'  --quickcheck-replay=485789
  7. It should always fail with the same random value