search

ctco / cukes

Cucumber DSL for testing RESTful Web Services
Apache License 2.0
112 stars 66 forks source link

Number of tests (total, failed, etc.) vary wildly #13

Open berndgoetz opened 8 years ago

berndgoetz commented 8 years ago

When running our tests, the reported numbers of test executions vary wildly. Here are some outcomes, based on the same code base, server and test client:

Tests run: 2622, Failures: 14, Errors: 0, Skipped: 8 Tests run: 2609, Failures: 4, Errors: 0, Skipped: 1 Tests run: 2302, Failures: 0, Errors: 404, Skipped: 1268 Tests run: 2623, Failures: 0, Errors: 0, Skipped: 0 Tests run: 2302, Failures: 0, Errors: 404, Skipped: 1268 Tests run: 2609, Failures: 4, Errors: 0, Skipped: 5

I haven't figured out the math here. Adding up the numbers don't match the overall number of tests. Any explanations? Any idea how to fix/improve this? Thanks.

AlexeyBuzdin commented 8 years ago

This appears to be a global issue with cucumber https://github.com/cucumber/cucumber-jvm/issues/577 and https://github.com/cucumber/cucumber-jvm/issues/263. Cucumber report is generated correctly though. Managed to create a workaround with TeamCity and Bamboo configurations. Will look into the issue further

AlexeyBuzdin commented 8 years ago

The surefire integration in Cucumber isn't that stable at all, so a possible solution is to create a stand-alone maven/gradle plugin and use cucumber.json as a source for reports. The .json itself is generated correctly and supported in Bamboo. The issue will be solved with https://github.com/ctco/cukes-rest/issues/45