deepbrook
commented
4 years ago Running one of the nightly scenarios seems to touch on too many components that would be out of scope for a smoke test.
Per definition, smoke tests should verify the build, indicating if any further testing makes sense.
That being said, I believe the following would be the desired scope of such tests:
- Node Setup works (we can fire up a number of raiden nodes for a scenario) - this includes looking up binaries, creating keystores and starting the processes
- Node funding works
- udc deposits work
- deploying custom tokens works
- configuring a pfs works
- tear down works
Luckily, this can be tested with a no-op scenario. Although this isn't quite ideal (it still requires an internet connection for contract deployments, etc), it would serve as a good start.
Dominik1999
Analysis
Current State
Currently, the test coverage is at ~60%. It is important to keep the scenario player stable. Therefore we are running a manual test if major changes have been made.
Problem
The manual tests are time-consuming and might not touch all parts of the software. In addition small changes might not get smoke tested and therefore lead to errors.
Solution
Create a pragmatic way (eg. running the bf1 scenario) for a smoke test