Tests executed by the developers easily and reproducibly are a requirement for reliable software development. There is a couple of
issues complicating the execution of the test suite.
Execution against preprod
The tests are executed on the preprod chain. This approach gives best idea on how the protocol behaves in the wild. The challenge is though to reproduce the tests, which for a developer is helpful to do often, even as often as on every validator change.
The protocol initialization mints a one-shot-mint NFT token by spending a specified UTXO. This means that on every NFT script change a new UTXO needs to be queried on blockchain, then hardcoded into the validator's source code, new validators compiled and then the testsuite can run. When we don't change the NFT, then the mintNFT action can be commented out. In other words many source code modifications need to be performed to run the tests with updated policies.
Fully applied scripts
The issue connects with the above. The plutus scripts get fully parametrized to yield the validator and minting policies already in the onchain code. As all the 3 scripts depend directly or indirectly on the UTXO to be spend in the one-shot-mint transaction - this means that the source code needs to be recompiled with queried information every time an NFT policy changes. This also forbids test execution in the emulator.
Unknown dependency versions
The MLabs team could not reproduce the build of the typescript offchain tests with the provided package versions. The team was able to build the testsuite using newer nodejs version (v20.11.1). The node packages used as project dependencies (so ts-node-esm, typescript) should be specified and their versions pinned in the tests/package.json file.
Recommendation
There exists many tools for testing plutus scripts, Plutip, PSM to name some opensource ones developed by MLabs. Every tool covers slightly different space and has its tradeoffs, therefore the codebase would benefit from integrating both.
Here we recommend to mirror the existing testsuite in the Lucid emulator and also MLabs provides a contribution implementing such a change. We choose this approach because it integrates easily into the existing codebase and is the most feasible to implement in the given time span. We provide the suggested implementation without guarantee of any kind and advice the SingularityNet team to review it thoroughly. At the added ergonomics of running the testsuite completely offline and reproducibly, there was introduced the complexity of applying the validator scripts dynamically in the offchain code. We note though, that the ability to apply scripts dynamically sooner or later becomes a necessity of the offchain application utilizing the validators.
The pr brings changes to the scripts allowing the offchain to parametrize them dynamically and defines a parallel test suite that runs against an emulator instead of preprod network. We discuss the details of the implementation in the pr.
Description
Tests executed by the developers easily and reproducibly are a requirement for reliable software development. There is a couple of issues complicating the execution of the test suite.
Execution against preprod
The tests are executed on the preprod chain. This approach gives best idea on how the protocol behaves in the wild. The challenge is though to reproduce the tests, which for a developer is helpful to do often, even as often as on every validator change.
The protocol initialization mints a one-shot-mint NFT token by spending a specified UTXO. This means that on every NFT script change a new UTXO needs to be queried on blockchain, then hardcoded into the validator's source code, new validators compiled and then the testsuite can run. When we don't change the NFT, then the
mintNFTaction can be commented out. In other words many source code modifications need to be performed to run the tests with updated policies.Fully applied scripts
The issue connects with the above. The plutus scripts get fully parametrized to yield the validator and minting policies already in the onchain code. As all the 3 scripts depend directly or indirectly on the UTXO to be spend in the one-shot-mint transaction - this means that the source code needs to be recompiled with queried information every time an NFT policy changes. This also forbids test execution in the emulator.
Unknown dependency versions
The MLabs team could not reproduce the build of the typescript offchain tests with the provided package versions. The team was able to build the testsuite using newer
nodejsversion (v20.11.1). The node packages used as project dependencies (sots-node-esm,typescript) should be specified and their versions pinned in thetests/package.jsonfile.Recommendation
There exists many tools for testing plutus scripts,
Plutip,PSMto name some opensource ones developed by MLabs. Every tool covers slightly different space and has its tradeoffs, therefore the codebase would benefit from integrating both.Here we recommend to mirror the existing testsuite in the
Lucidemulator and also MLabs provides a contribution implementing such a change. We choose this approach because it integrates easily into the existing codebase and is the most feasible to implement in the given time span. We provide the suggested implementation without guarantee of any kind and advice the SingularityNet team to review it thoroughly. At the added ergonomics of running the testsuite completely offline and reproducibly, there was introduced the complexity of applying the validator scripts dynamically in the offchain code. We note though, that the ability to apply scripts dynamically sooner or later becomes a necessity of the offchain application utilizing the validators.The pr brings changes to the scripts allowing the offchain to parametrize them dynamically and defines a parallel test suite that runs against an emulator instead of preprod network. We discuss the details of the implementation in the pr.