Cucumber report job being marked as failed

[mattharr]

I was just looking at this job in the Jenkins dashboard, and saw the last couple of runs had failed (red ball), when I had a look at them it looked like they had run, both from the console, and looking at the generated reports.

I've realised that if there are any failures in the report it marks the job as having failed, I'm not sure this seems right - I think the report worked, but it just reported errors.

Does this sound right to you?

I had a look, and I think its this line: buildResult = featureReportGenerator.getBuildStatus();

in the cucumber report publisher.

Do you want to look at this, or would you like me to create a pull request (I'm currently just getting my feet wet with GitHub, so don't mind having a go at this :) )

Thanks,

Matt

[kingsleyh]

Hi Matt,

The current behaviour is to fail the build if there are failures in the cucumber tests. This is to let us know if anything failed so we can fix them.

If you want test failures to cause the build to pass - this is not something I personally would want. But if it is useful to you and you want to create a pull request - please make this functionality optional by putting it in the config.

I hope I understood the issue correctly!

--k

Sent from my iPhone

On 31 May 2012, at 00:22, Mattreply@reply.github.com wrote:

I was just looking at this job in the Jenkins dashboard, and saw the last couple of runs had failed (red ball), when I had a look at them it looked like they had run, both from the console, and looking at the generated reports.

I've realised that if there are any failures in the report it marks the job as having failed, I'm not sure this seems right - I think the report worked, but it just reported errors.

Does this sound right to you?

I had a look, and I think its this line: buildResult = featureReportGenerator.getBuildStatus();

in the cucumber report publisher.

Do you want to look at this, or would you like me to create a pull request (I'm currently just getting my feet wet with GitHub, so don't mind having a go at this :) )

Thanks,

Matt

---

Reply to this email directly or view it on GitHub: https://github.com/masterthought/jenkins-cucumber-jvm-reports-plugin-java/issues/14

[mattharr]

Yes, I had a second think about this on the way home today - you're right it should fail the build I think. I'd had a bit of an odd case with this, in that my main build had succeeded (incorrectly), and then this build had failed, so it looked very odd - but that was because I'm fiddling around with trying to get the tests running in parallel - which gives the odd behaviour of running the tests, and then it tels you that no test ran! (so it regarded the build as passed).

Anyway - I agree with you :)

Thanks,

Matt

[surendrakashyap]

I think, this should be driven through property like surefire and failuresafe plug. Different firm have different meaning of FAILURE.

[kingsleyh]

I disagree about that - everyone should treat failing tests as failure. However although I disagree that a build should pass if there are failing tests - I do see there could be edge cases where other people may not want failing tests to fail the build - it's not something I would ever do in general but I will add an option to allow failures to not fail the build.

--K

On 9 May 2013, at 18:18, surendrakashyap notifications@github.com wrote:

I think, this should be driven through property like surefire and failuresafe plug. Different firm have different meaning of FAILURE.

— Reply to this email directly or view it on GitHub.

[surendrakashyap]

I understand, having the option is important just like sure fire or failsafe. I think, marking build unstable would be a better choice as opposed to failing.

Sent from my iPhone

On May 9, 2013, at 1:35 PM, Kingsley Hendrickse notifications@github.com wrote:

I disagree about that - everyone should treat failing tests as failure. However although I disagree that a build should pass if there are failing tests - I do see there could be edge cases where other people may not want failing tests to fail the build - it's not something I would ever do in general but I will add an option to allow failures to not fail the build.

--K

On 9 May 2013, at 18:18, surendrakashyap notifications@github.com wrote:

I think, this should be driven through property like surefire and failuresafe plug. Different firm have different meaning of FAILURE.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub.