Individual test errors not reflected in final gradle build status

[rsideserf]

When a gradle build is executed and tests run using courgette-jvm, an individual test may fail but this is not reflected in the overall build status that is still shown as successful.

I would expect the build to fail when any test fails.

[rsideserf]

Looks like an additional line may fix it in https://github.com/prashant-ramcharan/courgette-jvm/blob/8692eee327771e5b03ce428966049d13f47a00f7/src/main/java/courgette/api/junit/Courgette.java#L74

after line 74 by adding :

 CourgetteTestErrorException.throwTestErrorException();

[prashant-ramcharan]

Test failures are handled correctly and gradle does report the failure when tests fail as part of the Courgette run.

Can you share your build.gradle or confirm that you don't have ignoreFailures in your build.gradle ?

https://github.com/prashant-ramcharan/courgette-jvm/blob/master/src/main/java/courgette/runtime/junit/CourgetteJUnitRunner.java#L104

[rsideserf]

No sign of ignoreFailures anywhere in the build.gradle or project.

I think it may be that we are running a number of feature files in parallel using a single Cucumber @Regression tag that appears in more than one feature file. Only one of the feature files produces an error.

If I run that error producing feature file on it's own then the build fails as desired. However when I run them in parallel with both successful and error generating feature files, the build is successful when it should still fail.

Can you please check that scenario?

[prashant-ramcharan]

Using a single tag across multiple feature files would not cause this error.

I'm not able to reproduce this. Do you have a demo project I can use to replicate this?

You can try with the Courgette demo project if you like: https://github.com/prashant-ramcharan/courgette-jvm-example

[rsideserf]

Thank you. I will try to recreate with the demo project and get back to you.

[rsideserf]

I have been unable to recreate our issue using the example program. Our project is obviously more complex and has multiple modules with corresponding build.gradle files. I have tried to reproduce a similar structure in the example but have been unable to reproduce. I cannot spend any more time on this so have implemented the fix that works for us above and that other projects in our organisation have had success with.

I'll close the issue for now and investigate again when there is more time.

Thanks for your help.

[prashant-ramcharan]

As you using a multi module gradle project, you could implement a ProjectEvaluationListener to evaluate the failure but I'd imagine that if your gradle tasks are configured correctly, you wouldn't have too.

Project.afterEvaluate()

https://docs.gradle.org/current/userguide/build_lifecycle.html#build_lifecycle_events

Having said that, I've been using Courgette in a multi module Gradle project without any issues.