Now VSharp and TestRunner will return a null code even if some of the tests fail, or if exceptions are thrown during branch parsing.
I propose to consider the possibility of returning different codes depending on the quality of the work done. For example, if most of the generated tests fail, then you can return a special code. Also, the codes may depend on the degree of importance of the detected errors.
The proposed approach will make it more convenient to embed VSharp in CI systems - automation will be able to interpret the results in a single way. In particular, it will be easier for us to understand with what status the benchmarks ended.
Now VSharp and TestRunner will return a null code even if some of the tests fail, or if exceptions are thrown during branch parsing.
I propose to consider the possibility of returning different codes depending on the quality of the work done. For example, if most of the generated tests fail, then you can return a special code. Also, the codes may depend on the degree of importance of the detected errors.
An example of such a system can be found here: https://pvs-studio.com/en/docs/manual/0035/.
The proposed approach will make it more convenient to embed VSharp in CI systems - automation will be able to interpret the results in a single way. In particular, it will be easier for us to understand with what status the benchmarks ended.