search

jenkinsci / testrail-plugin

Integrate test results from Jenkins into TestRail
Apache License 2.0
45 stars 35 forks source link

Allow multiple executions of a single case to be uploaded separately #45

Open systemsincode opened 5 years ago

systemsincode commented 5 years ago

Hi we are using the Jenkins test rail plugin and we are getting false positives for test cases from scenario outlines.

The current implementation logs scenario outline example executions as multiple executions of the same test case in the same run. If the last example to run passed then the test case is passed, even if all other examples actually failed.

Has anyone experienced this behaviour and did you find a way to change it so if Any fail then the test case is failed or to list each example execution as a different test case?

xirsoi commented 5 years ago

I don't understand what you mean by example executions? Why are you executing the same test multiple times in a single run?

systemsincode commented 5 years ago

In Cucumber each scenario outline is re-executed for each line of the Example table. But it typically has the same scenario name as variables vlaues are typically used in steps only. As this plugin doesn't use ids and scenario names only (as far as I can gleam) so as far as the TestRail is concerned the Scenario names are not changed, therefore they have simply been executed multiple times in each outline. This is only a problem when some may fail but not all as if the last to execute passes it will mark the test as a pass for a Run. This means failures within a scenario outline can get 'masked' if you are not aware of the issue.

As a simple workaround I have found you can simply add id column and refernce that in your scenario outline name. See https://github.com/systemsincode/cucumber-java-skeleton-TestRail/blob/master/src/test/resources/io/cucumber/skeleton/TestRailTest1.feature line 62 for example. I was hoping there was a flag for the plugin to do this for you somehow, but I guess not?

xirsoi commented 5 years ago

Ahh, I see. I don't have much experience with Gherkin/Cucumber, so I hadn't considered those scenarios.

Clever workaround!

No, there's no toggle like that, but it's a good idea. I'll see if I can work that behavior in.

xirsoi commented 5 years ago

Note: Will have to determine an order of precedence for the various result states. Obviously failures should be uploaded after passes, but should an error overwrite a failure?

systemsincode commented 5 years ago

hmm not sure I think the fix should be to unique-ify rather than change the behaiour for executions of the same test cases. I think Scenario outlines are a 'special' case in that they are 'meant' to be different in some way - i.e. if each example row differs I think it is a different kind of test than runnig the same test with the same data twice. I guess the pluggin would have to 'know a test is from an outline' but I don't think there is enought information in the XML to determine that.

On the other hand failing if any executions within a run for the same test would by pass my issue, but I'm trying to think of why you might want to repeat tests untill they pass and it does thenits ok and this would break this working as per now? I can work around it anyway...