search

Boehringer-Ingelheim / toscaci

tosca command line tool and service that simplify tosca integration with Continuous Integration tools
Apache License 2.0
10 stars 7 forks source link

Tosca CI / toscactl returns wrong errorcode on failed / errored tests #56

Open clemensutschig opened 2 years ago

clemensutschig commented 2 years ago

It seems some? tosca test failures are translated into errors in xUnit files- and then the exit return code generated is wrongly set to 0 which is wrong

https://github.com/Boehringer-Ingelheim/toscaci/blob/6239ea8fd4ea92303202217c428a871aade2fdbf/client/src/go/toscactl/cli/run.go#L145-L153

from the logs of tosca / CIService:

DEBUG: Reporting unit test results to corresponding test cases in Jira. Test type: '[Acceptance]'.
Test results: [testsuites:[[....., failures:0, tests:46, name:Acceptance, id:d2a6a6af-1466-43a1-8c44-b284101ea538, time:16237, errors:2, timestamp:2/19/2022 10:16:55 PM, skipped:0, properties:[], testcases:[[classname:Acceptance, name:Check Settings used in Execution, time:0, skipped:false, systemOut:  |

While the report shows:

image

m-lipovac commented 2 years ago

I think you will be glad to hear that this feature has literally been checked into the Tosca 15.1 product 2 weeks ago 😃 The official release of 15.1 will be in a week or 2 if I am not mistaken. The error and failure tags will now contain the logs from the first failing TestStep.