search

selenide / selenide

Concise UI Tests with Java!
http://selenide.org
MIT License
1.8k stars 566 forks source link

Soft Assert mode with a FAIL step is not marking the whole test as FAILED #2755

Open jensenbrian opened 1 month ago

jensenbrian commented 1 month ago

The problem

In Soft Assert mode with a FAIL step is not marking the whole test as FAILED

Details

Running the TestNgSoftTest.textArea() in SOFT mode with an added line that will fail. The build is marked as "BUILD SUCCESSFUL". (see branch testng-softassert-not-failing HERE)

Running the same change made to TestNgSampleTest.textArea() in STRICT mode. The test does stop processing and marks the build as "BUILD FAILED". (This is not an issue. Working as expected.)

Tell us about your environment

Code To Reproduce Issue

modules/testng/src/test/java/integration/TestNgSoftTest.java

  @Override
  @BeforeSuite
  final void setupAsserts() {
    Configuration.assertionMode = SOFT;
    SoftAsserts.fullStacktraces = false;
    closeWebDriver();
    WebDriverRunner.addListener(listener1);
  }

  @Test
  public void textAreaShouldFail() {
    $("#empty-text-area").val("text for textarea");
    // Will fail (on purpose) test will continue but should be marked as a FAILED test is in SOFT mode
    $("#empty-text-area").shouldNotHave(value("text for textarea"));
    $("#empty-text-area").shouldHave(value("text for textarea"));
  }
jensenbrian commented 1 month ago

I found in the changelog: 6.6.0 (released 08.06.2022)

1834 fix SoftAsserts with TestNG (downgraded TestNG from 7.5 to 7.4.0) -- see PR #1843

Looks like with the 7.0.0 release - bump TestNG from 7.4.0 to 7.8.0 (#2515)

I recently updated from 6.19.1 to 7.3.2 Could this still be broken in TestNG? If so should TestNG be downgraded back to 7.4.0?

I will see if I can figure out how to try that. Thank you for your thoughts/review

BorisOsipov commented 1 month ago

related to https://github.com/selenide/selenide/pull/1843 https://github.com/testng-team/testng/issues/2771 https://github.com/testng-team/testng/issues/3046

jensenbrian commented 1 month ago

Thanks @BorisOsipov . Looks like the "right" fix mentioned in #3046 is to use TestNG 7.10.0 and "Ordering listeners in TestNG" documented in Specifying listeners with ServiceLoader

I will see what I can do. "I am working on it"