Generating runtime/intermediate test reports

[nerissad]

When we run our entire test suite, consisting of multiple feature files, the test report files (cucumber format) get generated only after the entire run. For any reason, should the test suite build get killed, we get no report. To solve this, we were wondering if there is a way to incremental write/overrite the test report after each feature, so we can get intermiate reports, as well as have some results in case of build kill etc.

Would it be feasible to have for example from CucumberJSONWriterEachFeature instead of using after.all to use after.each_feature instead ? (with relevant changes to generate_ccjson)

[fliiiix]

For any reason, should the test suite build get killed, we get no report.

I mean yeah, but is that an actual use-case? Guessing from this issue this happens to you?

To solve this, we were wondering if there is a way to incremental write/overwrite the test report after each feature, so we can get intermediate reports, as well as have some results in case of build kill etc.

If that's a real thing i would propose a after feature writer which creates a result for each feature and keep the current write untouched.

[nerissad]

For any reason, should the test suite build get killed, we get no report.

I mean yeah, but is that an actual use-case? Guessing from this issue this happens to you?

To solve this, we were wondering if there is a way to incremental write/overwrite the test report after each feature, so we can get intermediate reports, as well as have some results in case of build kill etc.

If that's a real thing i would propose a after feature writer which creates a result for each feature and keep the current write untouched.

yes this happens, since our CI pipelines also have a timeout should for any reason the tests take too long ( maybe we don't have fail fast designed the best way inside the steps, so it adds up, this is one of the possibiliies ). As an end result, we end up with no report, but just log files. I can also call radish itself multiple times instead of just specifying the whole testsuite. This could be a another way to solve it.

[nerissad]

Looking at an example from a branch on this repo, I was able to solve my specific problem (i.e of kill process vs keyboard interrupt which is handled gracefully already in radish) by these lines of code with terrrain/hooks.

#To be able to generate cucumber reports after each feature execution so that the whole run is not lost when the process is killed.
@after.each_feature
def generate_cucumber_report(feature):
    if world.config.cucumber_json:
        cjw = CucumberJSONWriter()
        cjw.generate_ccjson([feature], None)

[fliiiix]

I think i would be ok if we are aborted to make sure the CucumberJSONWriter is called.

[fliiiix]

@nerissad are you up to proposing a fix for that?

[nerissad]

@fliiiix what worked for me was the above code, which I put in my terrain.py file. This would make it a user configurable way, instead of being part of the radish codebase.

From the radish code base, I see it's already handled for a user abort to generate the cucumber report (example KeyboardInterrupt). however if you find the process and kill it, this is not handled (SIGTERM??). Are you suggesting to add handling for this directly in radish ?

Or to have a CucumberJSONWriter function which works after.each_feature instead of after.all ?

[fliiiix]

Are you suggesting to add handling for this directly in radish ?

Yeah that was kinda the idea i think the only case where no clean shutdown should happen is on SIGKILL. I think it would be fair that we try to handle that the same way as a KeyboardInterrupt