search

gradle / test-retry-gradle-plugin

Gradle plugin to retry tests that have failed to mitigate test flakiness.
Apache License 2.0
224 stars 50 forks source link

unable to retry the following test methods, which is unexpected #127

Closed Bersikan closed 5 months ago

Bersikan commented 2 years ago

every build i have an issue: [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception. [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] What went wrong: [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Execution failed for task ':runSmokeTests'. [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > org.gradle.test-retry was unable to retry the following test methods, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] com.vaultmr.qaauto.tests.patient.profile.tests.flowSheets.neuroChecks.NeuroChecksTests#fillAllDataAndSignChekVitalsAndRibbon [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] com.vaultmr.qaauto.tests.patient.profile.tests.CreateTests#validateSameMRNPatientTestDischargedPatient [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] com.vaultmr.qaauto.tests.patient.profile.tests.nutrition.NutritionTest#nutritionFollowUpTaskTest [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] com.vaultmr.qaauto.tests.charges.ChargesTests#modifyChargesInterfaceAndFlow [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] com.vaultmr.qaauto.tests.patient.profile.tests.cpoe.CPOE_eMAR_Tests_v2#diagnosisInteractions [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] com.vaultmr.qaauto.tests.facilityTemplates.tests.OrderSetsTests#createSingleOrderSetAncillaryOrderOnly [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Try: [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Run with --stacktrace option to get the stack trace. [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Run with --scan to get full insights. [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Get more help at https://help.gradle.org [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildResultLogger] [19:47:07]W: [Gradle failure report] 2022-02-21T11:47:07.003-0600 [ERROR] [org.gradle.internal.buildevents.BuildResultLogger] BUILD FAILED in 3h 12m 57s

list of test cases that was unable to retry is always different

for example 1st test case of this list failed with Element not found {Vitals} Expected: visible or transparent: visible or have css value opacity=0 Screenshot: file:/C:/BuildAgent/work/4bd6faa4a136f939/vaultmr-automation-tests/build/reports/tests/1645462580415.17.png Page source: file:/C:/BuildAgent/work/4bd6faa4a136f939/vaultmr-automation-tests/build/reports/tests/1645462580415.17.html Timeout: 10 s. Caused by: StaleElementReferenceException: stale element reference: element is not attached to the page document

env: gradle 7.3.2 Selenide 6.3.1 java 8 TestNG 7.4.0 id 'org.gradle.test-retry' version '1.3.1'

marcphilipp commented 2 years ago

@Bersikan Could you please provide a reproducer project?

Bersikan commented 2 years ago

Sorry, but no, we have security restrictions. i can attach full log files with debug mode. I just can say i cant reproduce it locally with single thread, only on CI in several threads (8)

Bersikan commented 2 years ago

Vaultmr_Test_Smoke_Suit_1662.log Vaultmr_Test_Smoke_Suit_1661.log Vaultmr_Test_Smoke_Suit_1664.log Vaultmr_Test_Smoke_Suit_1663.log

pshevche commented 5 months ago

Unfortunately, we won't be able to proceed without some form of a reproducer. A class skeleton of failing tests would help. I would also recommend trying the latest plugin version. Versions 1.4.x and 1.5.x had multiple improvements for retrying flaky TestNG tests and lifecycle methods.