courgette.runtime.CourgetteTestErrorException error with all cucumber tests passed

[AnnGur]

Hi Prashant, I've updated courgette-jvm version from 3.3.0 to 6.4.2 and I obtain following error from Courgette after cucumber tests execution execution (Gradle + JUnit).

1 Scenarios (1 passed)
7 Steps (7 passed)
14m56.726s

package.CucumberTest > executionError FAILED
    courgette.runtime.CourgetteTestErrorException

2 tests completed, 1 failed

> Task :runTest FAILED

Plugins used: id 'org.springframework.boot' version '2.6.7'

Dependencies used: implementation "io.cucumber:cucumber-spring:7.3.4" implementation 'io.cucumber:cucumber-java8:7.3.3'

Gradle version: '7.4.2'`

Features runner is as follows:

import courgette.api.CourgetteBeforeAll;
import courgette.api.CourgetteOptions;
import courgette.api.CourgetteRunLevel;
import courgette.api.CucumberOptions;
import courgette.api.junit.Courgette;
import courgette.runtime.event.CourgetteEvent;
import org.junit.runner.RunWith;

@RunWith(Courgette.class)
@CourgetteOptions(
        threads = 1,
        runLevel = CourgetteRunLevel.FEATURE,
        rerunFailedScenarios = false,
        rerunAttempts = 1,
        showTestOutput = true,
        reportTargetDir = "build",
        slackWebhookUrl = "https://hooks.slack.com/services/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",
        slackChannel = {"regression-test-reports"},
        slackTestId = "Customer Query",
        slackEventSubscription = {CourgetteEvent.TEST_FAILED, CourgetteEvent.TEST_PASSED, CourgetteEvent.TEST_RUN_SUMMARY},
        cucumberOptions = @CucumberOptions(
                glue = {"com.project"},
                tags = {"@report"},
                plugin = {"pretty", "json:build/cucumber.json", "html:build/cucumber-report"},
                features = {"src/test/resources/features"}
        )
)
public final class CucumberTest {
}

Thanks in advance for any help!

[prashant-ramcharan]

Hi @AnnGur

1. Can you please provide the Courgette or Cucumber reports from your build directory?
2. Does this happen on every run?

[AnnGur]

1. Courgette report page is here: https://drive.google.com/file/d/1RjUoZ8FaNjnerjY-ruVySRTA_24281P5/view?usp=sharing
2. Yes, it's happening every time, for test case which is otherwise green.

[prashant-ramcharan]

@AnnGur I dont have access to the google drive file.

Please attach it here or email.

Thanks

[AnnGur]

Done!

[prashant-ramcharan]

The report shows that the tests ran in 1 second which seems odd and the results are not in the table.

How do you know that all tests have passed if the report shows a failure as well as gradle?

Please provide the full gradle output or provide a project to replicate this issue.

Does the Courgette example project work fine ?

[AnnGur]

I know that the tests are passed because I see corresponding log in console:

1 Scenarios (1 passed)
7 Steps (7 passed)
14m56.726s

For Courgette example project - I've run "gradle runFeaturesUsingJUnit" and I've got a valid report:

For logs 2 Scenarios (1 failed, 1 passed) 4 Steps (1 failed, 3 passed) 0m10.195s

As for the full gradle output, I'll provide it a bit later, as it requires some time to generate it and mask project data. I'll be back with it soon, thanks for tracking!

[prashant-ramcharan]

This just suggests that one of the scenarios passed. If you running more than 1 scenario it could be that another scenario has failed.

1 Scenarios (1 passed)
7 Steps (7 passed)

So the full gradle output should show the failing scenario(s).

[AnnGur]

Nope, I'me running just 1 scenario for this smoke test. Tab declared to be run (tags = {"@report"}) exists at 1 test only. Here's the stacktrace:

sh-3.2$ gradle runTest
> Task :runTest
20:22:36,603 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback-test.xml] at [file:/Users/.../logback-test.xml]
20:22:36,632 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
20:22:36,635 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
20:22:36,638 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
20:22:36,667 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - This appender no longer admits a layout as a sub-component, set an encoder instead.
20:22:36,667 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder.
20:22:36,667 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details
20:22:36,667 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
20:22:36,667 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
20:22:36,667 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
20:22:36,668 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@43ed0ff3 - Registering current configuration as safe fallback point

2022-05-23 20:22:36,760  INFO [main] -- o.s.t.c.support.AbstractContextLoader    {}: Could not detect default resource locations for test class [<package>.SetupSteps]: no resource found for suffixes {-context.xml, Context.groovy}.
2022-05-23 20:22:36,982  INFO [main] -- .b.t.c.SpringBootTestContextBootstrapper {}: Loaded default TestExecutionListener class names from location [META-INF/spring.factories]: [org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener, org.springframework.boot.test.autoconfigure.webservices.client.MockWebServiceServerTestExecutionListener, org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener, org.springframework.test.context.web.ServletTestExecutionListener, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener, org.springframework.test.context.event.ApplicationEventsTestExecutionListener, org.springframework.test.context.support.DependencyInjectionTestExecutionListener, org.springframework.test.context.support.DirtiesContextTestExecutionListener, org.springframework.test.context.transaction.TransactionalTestExecutionListener, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener, org.springframework.test.context.event.EventPublishingTestExecutionListener]
2022-05-23 20:22:37,007  INFO [main] -- .b.t.c.SpringBootTestContextBootstrapper {}: Using TestExecutionListeners: [org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener@4bdc8b5d, org.springframework.test.context.event.ApplicationEventsTestExecutionListener@3bcd426c, org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener@5f14a673, org.springframework.boot.test.autoconfigure.SpringBootDependencyInjectionTestExecutionListener@726a17c4, org.springframework.test.context.support.DirtiesContextTestExecutionListener@5dc3fcb7, org.springframework.test.context.transaction.TransactionalTestExecutionListener@c4c0b41, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener@76911385, org.springframework.test.context.event.EventPublishingTestExecutionListener@5467eea4, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener@160396db, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener@7a799159, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener@40ab8a8, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener@6ff37443, org.springframework.boot.test.autoconfigure.webservices.client.MockWebServiceServerTestExecutionListener@65cc8228, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener@53093491]

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.6.7)

2022-05-23 20:22:38,210  INFO [main] -- .s.d.r.c.RepositoryConfigurationDelegate {}: Bootstrapping Spring Data JDBC repositories in DEFAULT mode.
2022-05-23 20:22:38,243  INFO [main] -- .s.d.r.c.RepositoryConfigurationDelegate {}: Finished Spring Data repository scanning in 27 ms. Found 0 JDBC repository interfaces.

<...>

@report @Negative @Regression
Scenario: Case 2: Query                                                                                                                # src/test/resources/features/query_negative_tests.feature:8
  <steps>

1 Scenarios (1 passed)ompleted
7 Steps (7 passed)
12m1.320s

<package>.CucumberTest > executionError FAILED
    courgette.runtime.CourgetteTestErrorException

2 tests completed, 1 failed

> Task :runTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':runTest'.
> There were failing tests. See the report at: file:///Users/<...>/build/reports/tests/runTest/index.html

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 11s
5 actionable tasks: 4 executed, 1 up-to-date

I've updated report (from build folder) via link as well.

[prashant-ramcharan]

Courgette 6.4.2 only supports Cucumber version 7.2.3 so there could be an issue at runtime when using a higher Cucumber version (but not verified)

Can you try updating your dependencies in your project to match:

Update the Cucumber Spring dependancy version

    implementation 'io.cucumber:cucumber-spring:7.2.3'      // <--- update to this
    implementation 'io.github.prashant-ramcharan:courgette-jvm:6.4.2'

Remove any other Cucumber dependancy

    implementation 'io.cucumber:cucumber-java8:7.3.3' // <--- remove this

[AnnGur]

Having same issue :(

1 Scenarios (1 passed)
7 Steps (7 passed)
12m20.591s

package.CucumberTest > executionError FAILED
    courgette.runtime.CourgetteTestErrorException

2 tests completed, 1 failed

> Task :runTest FAILED

And I've also added integration with slack and here's the notification after this run: :white_check_mark: Test Passed Timestamp: 2022-05-24T08:52:33.103815Z Session Id: 59f87c38af6349bca47ef0aceecfe9a1 Test Id: Query Feature: query_negative_tests

:information_source: Test Run Summary Timestamp: 2022-05-24T08:52:34.290188Z Session Id: 59f87c38af6349bca47ef0aceecfe9a1 Test Id: Customer Query Run Duration: 12 min, 22 sec Run Summary: 100% passed, 0% failed Total Features: 1 Passed: 1 Failed: 0

My build.gradle is as follows:

buildscript {
    ext {
        springbootVersion = '2.6.7'
        awsJavaSdkVersion = '1.11.801'
    }
    repositories {
        mavenCentral()
    }
}

plugins {
    id 'org.springframework.boot' version '2.6.7'
    id 'io.freefair.lombok' version '6.4.3'
}

apply plugin: 'java'
apply plugin: 'io.spring.dependency-management'
apply plugin: 'idea'
apply plugin: 'application'

sourceCompatibility = 1.8
mainClassName = "package.CucumberTest"

def bams_url = 
def bams_user = 
def bams_pass = 

repositories {
    mavenCentral()
    maven {...}
}

configurations.all {
    exclude group: "org.slf4j", module: "slf4j-log4j12"
    exclude group: "org.apache.logging.log4j", module: "log4j-slf4j"
    exclude group: "org.apache.logging.log4j", module: "log4j-slf4j-impl"
    exclude group: "org.apache.logging.log4j", module: "log4j-api"
    exclude group: "org.apache.logging.log4j", module: "log4j-to-slf4j"
}

dependencies {
    implementation "org.springframework.boot:spring-boot-dependencies:$springbootVersion"
    implementation 'org.springframework:spring-tx'
    implementation('org.springframework.boot:spring-boot-starter-test') {
        exclude group: "com.vaadin.external.google", module: "android-json"
    }
    implementation "org.springframework.boot:spring-boot-configuration-processor"
    implementation 'org.springframework.boot:spring-boot-starter-data-jdbc'

    implementation group: 'com.amazonaws', name: 'aws-java-sdk', version: awsJavaSdkVersion
    implementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.12.0'

    implementation 'com.oracle.database.jdbc:ucp:19.6.0.0'
    implementation group: 'com.oracle.database.jdbc', name: 'ojdbc8', version: '19.8.0.0'

    implementation "ch.qos.logback:logback-core:1.2.3"

    implementation "com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.13.3"
    implementation "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.13.3"

    implementation 'io.rest-assured:rest-assured:3.3.0'

    implementation group: 'org.assertj', name: 'assertj-core', version: '3.16.1'
    implementation 'io.github.prashant-ramcharan:courgette-jvm:6.4.2'
    implementation "io.cucumber:cucumber-spring:7.2.3"
//    implementation 'io.cucumber:cucumber-java8:7.3.3'
    implementation 'io.cucumber:cucumber-junit:7.3.3'
    implementation group: 'org.awaitility', name: 'awaitility', version: '3.0.0'
    implementation group: 'org.projectlombok', name: 'lombok', version: '1.18.24'
}

tasks.withType(Test) {
    systemProperties = System.getProperties()
    scanForTestClasses = false
}
task runTest(type: Test) {
    include '**/CucumberTest.class'
    outputs.upToDateWhen { false }
}

gradle.startParameter.continueOnFailure = true

wrapper {
    gradleVersion = '7.4.2'
    distributionUrl = distributionUrl.replace("bin", "all")
}

[AnnGur]

Hmmm - I've commented 'io.cucumber:cucumber-junit:7.3.3' as well and it is green!! I'll perform a Jenkins run and come back with result.

[prashant-ramcharan]

Great! Let me know.

I'm also going to add a bit more logging to the upcoming release so it's easier to identify errors in the test output.

Note: You don't need to add any additional cucumber dependencies (except Cucumber spring) when using Courgette because Courgette already includes the Cucumber deps.

[AnnGur]

Yeah, it works now!! Thanks a lot for your help!!