Running plutip tests take ages and lengthens the feedback loop.
It is proven to be possible to execute tests in parallel, however currently it require forking CTL and altering some parts of the framework.
Testing framework
purescript-spec is a good candidate:
It is very similar to mote, and it is able to run test cases in parallel, with flexible chunking.
However it's SpecT is a bit less general than MoteT and require altering, but that's not a big issue, and a small contribution to the library can easily fix the issue.
Parallelizing
Each testing instance should have it's own PlutipConfig, with non duplicating ports. This way we ensure they don't interfere.
Kupo workdirs will not interfere as Kupo's workdir contain a randomly generated number.
Ogmios and Plutip workirs interference have not proven to exist.
Importantly, graceful shutdown must be properly implemented so no matter what all temp directories must be cleaned up, and all the services must be shut down after tests.
Design
It is probably not the best idea to left the implementation in the CTL itself, but instead put it in the separate library.
To leave it CTL:
testnet cluster creation and running tools
contract for redistributing funds between tests
To separate:
MoteT/SpecT manipulation, creation, configuring and running a test tree.
Potential problems
Decreased cardano-node/ogmios/kupo performance on multiple parallel instances
Accidental errors will occur more often because of the CPU, Mem and disk usage.
Some operations may take longer to execute, and it should be concerned.
So for epoch-dependent (the ones that should be finished in a certain amount of epochs) operations increasing amount of parallel instances could require increasing the slot or epoch length.
Setting up tests to run in parallel may be complicated
This is not a big issue because
Parallel testing is optional
It is still better than enforcing every project that need this feature to alter part of the CTL codebase on every project and implement it themselves
Summary
Running plutip tests take ages and lengthens the feedback loop. It is proven to be possible to execute tests in parallel, however currently it require forking CTL and altering some parts of the framework.
Testing framework
purescript-specis a good candidate: It is very similar tomote, and it is able to run test cases in parallel, with flexible chunking. However it'sSpecTis a bit less general thanMoteTand require altering, but that's not a big issue, and a small contribution to the library can easily fix the issue.Parallelizing
Each testing instance should have it's own PlutipConfig, with non duplicating ports. This way we ensure they don't interfere. Kupo workdirs will not interfere as Kupo's workdir contain a randomly generated number. Ogmios and Plutip workirs interference have not proven to exist.
Importantly, graceful shutdown must be properly implemented so no matter what all temp directories must be cleaned up, and all the services must be shut down after tests.
Design
It is probably not the best idea to left the implementation in the CTL itself, but instead put it in the separate library.
To leave it CTL:
Potential problems
Decreased cardano-node/ogmios/kupo performance on multiple parallel instances
Accidental errors will occur more often because of the CPU, Mem and disk usage. Some operations may take longer to execute, and it should be concerned. So for epoch-dependent (the ones that should be finished in a certain amount of epochs) operations increasing amount of parallel instances could require increasing the slot or epoch length.
Setting up tests to run in parallel may be complicated
This is not a big issue because