Regression: Collected assertions generate duplicate screenshots

conradstrassburger commented 9 months ago

Describe the bug Currently, the report-ng-tests generate 1 failure.

To Reproduce Clone the repository and run gradle buildReport in report-ng. Then run gradle test -P pretest=true, gradle test -P pretestExtended=true and gradle test in report-ng-tests.

Expected behavior All tests pass.

Actual behavior One failure:

22.01.2024 15:25:11.738 [TestNG-PoolService-0][INFO][MCID:146]: start.MethodStartWorker - Run Test (ExclusiveSessionsTest.test_singleScreenshotShown())

Expected that LastScreenshotOverlay -> overlayContainer -> nextScreenshotButton displayed is false
eu.tsystems.mms.tic.testframework.exceptions.UiElementAssertionError: Expected that LastScreenshotOverlay -> overlayContainer -> nextScreenshotButton displayed is false
    at app//eu.tsystems.mms.tic.testframework.pageobjects.internal.asserts.DefaultUiElementAssertion$UiElementAssertionProvider.wrapAssertionError(DefaultUiElementAssertion.java:67)
    at app//eu.tsystems.mms.tic.testframework.internal.asserts.AbstractPropertyAssertion.wrapAssertionErrorRecursive(AbstractPropertyAssertion.java:106)
    at app//eu.tsystems.mms.tic.testframework.internal.asserts.AbstractTestedPropertyAssertion.testSequence(AbstractTestedPropertyAssertion.java:120)
    at app//eu.tsystems.mms.tic.testframework.internal.asserts.DefaultBinaryAssertion.is(DefaultBinaryAssertion.java:50)
    at app//eu.tsystems.mms.tic.testframework.internal.asserts.BinaryAssertion.is(BinaryAssertion.java:30)
    at app//eu.tsystems.mms.tic.testframework.pageobjects.internal.asserts.UiElementBaseAssertion.displayed(UiElementBaseAssertion.java:46)
    at app//io.testerra.report.test.pages.report.methodReport.LastScreenshotOverlay.assertSingleScreenshot(LastScreenshotOverlay.java:33)
    at app//io.testerra.report.test.report_test.common.ExclusiveSessionsTest.test_singleScreenshotShown(ExclusiveSessionsTest.java:61)
    at java.base@11.0.12/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base@11.0.12/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base@11.0.12/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.12/java.lang.reflect.Method.invoke(Method.java:566)
    at app//org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
    at app//org.testng.internal.invokers.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:258)
    at app//eu.tsystems.mms.tic.testframework.report.hooks.TestMethodHook.runHook(TestMethodHook.java:61)
    at app//eu.tsystems.mms.tic.testframework.report.TesterraListener.run(TesterraListener.java:379)
    at app//org.testng.internal.invokers.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:272)
    at app//org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:656)
    at app//org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:227)
    at app//org.testng.internal.invokers.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:80)
    at app//org.testng.internal.invokers.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:17)
    at java.base@11.0.12/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base@11.0.12/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base@11.0.12/java.lang.Thread.run(Thread.java:829)

Test https://github.com/telekom/testerra/blob/ecba64f735472651e524fbe490dfe7d9e1cca174/report-ng-tests/src/test/java/io/testerra/report/test/report_test/common/ExclusiveSessionsTest.java#L51-L62

This specific method seems to generate two screenshots instead of the expected one: https://github.com/telekom/testerra/blob/ecba64f735472651e524fbe490dfe7d9e1cca174/report-ng-tests/src/test/java/io/testerra/report/test/pretest_status/simple/GenerateScreenshotsInTesterraReportTest.java#L48-L57

Environment(please complete the following information): Testerra::Latest

Additional context Behaviour has been tested on 2.7 and 2.6.

martingrossmann commented 9 months ago

Hi, this is by design but unfortunately the report test wasn't updated:

After every assert failure of a collected assertion a screenshot is generaed to keep the current state (since Testerra 2.4)
After every failed method another screenshot will take.
Of course in that case it is not really useful but after a collected assertion a normal assertion could follow. Therefore its better to have to much screenshots as to less.

martingrossmann commented 9 months ago

@clemensgrosse Please adjust the testcase.

telekom / testerra

Regression: Collected assertions generate duplicate screenshots #390