bmr-cymru
commented
3 months ago I wonder if we could use a custom decorator here and avoid the need to pass the test method name explicitly? I.e. using __name__ from the decorator and then passing the result to skipIf? I haven't tried this out but it might allow us to avoid the need to pass the name and "skip requested" string explicitly. It's a small saving but if it works out it might be worth it.
mulkieran
The problem I'm addressing is that of not knowing what tests you will want to skip when you are writing the code.
We do some legacy tests on nightly. These legacy tests run older versions of the CLI against the current version of stratisd to try to detect if some failure has crept in. But sometimes, the tests are too precise, and FAIL with the new version of stratisd even when we consider the stratis-cli behavior to be correct. This can happen if, for example, we change the exception that is raised from StratisCliEngineError to StratisCliUserError, that is, we run a pre-check in stratis-cli of what we previously expected the engine to return an error for. In both cases, stratis-cli will exit with an error, and we consider this acceptable.
This approach will put in place a framework that will work for skipping any test that is desired to skip in future. It will be necessary to annotate every test with the skipIf decorator now and for future tests so that this approach can work in future.