search

mikepenz / action-junit-report

Reports junit test results as GitHub Pull Request Check
https://blog.mikepenz.dev
Apache License 2.0
298 stars 114 forks source link

Duplicate tests? #1136

Closed swwol closed 1 month ago

swwol commented 1 month ago

I'm trying this action on a Android repo - I'm seeing duplicate annotations for my failed test - and the total number of tests is twice what it should be - so it seems the tests are being counted twice. I've checked the xml files and these do not contain duplicates, so I'm guessing there is something about the format of the files that is leading to double counting.

Annotatons:

Screenshot 2024-07-15 at 14 22 29

This is the xml for the failed test:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="com.marksandspencer.home.component.banner.ui.TextBannerComponentTest" tests="6" skipped="0" failures="1" errors="0" timestamp="2024-07-12T12:44:42" hostname="runner" time="38.604">
  <properties/>
  <testcase name="this test should deliberately fail" classname="com.marksandspencer.home.component.banner.ui.TextBannerComponentTest" time="30.274">
    <failure message="java.lang.RuntimeException: This test is designed to fail" type="java.lang.RuntimeException">java.lang.RuntimeException: This test is designed to fail
    at com.marksandspencer.home.component.banner.ui.TextBannerComponentTest.this test should deliberately fail(TextBannerComponentTest.kt:28)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
    at androidx.compose.ui.test.junit4.AndroidComposeTestRule$apply$1$evaluate$1.invoke(AndroidComposeTestRule.android.kt:272)
    at androidx.compose.ui.test.junit4.AndroidComposeTestRule$apply$1$evaluate$1.invoke(AndroidComposeTestRule.android.kt:271)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment$AndroidComposeUiTestImpl.withDisposableContent(ComposeUiTest.android.kt:512)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment$runTest$1$1$1$1$1$1$1.invoke(ComposeUiTest.android.kt:330)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment.withTextInputService(ComposeUiTest.android.kt:396)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment.access$withTextInputService(ComposeUiTest.android.kt:234)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment$runTest$1$1$1$1$1$1.invoke(ComposeUiTest.android.kt:329)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment.withComposeIdlingResource(ComposeUiTest.android.kt:383)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment.access$withComposeIdlingResource(ComposeUiTest.android.kt:234)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment$runTest$1$1$1$1$1.invoke(ComposeUiTest.android.kt:328)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment.withWindowRecomposer(ComposeUiTest.android.kt:357)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment.access$withWindowRecomposer(ComposeUiTest.android.kt:234)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment$runTest$1$1$1$1.invoke(ComposeUiTest.android.kt:327)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment.withTestCoroutines(ComposeUiTest.android.kt:370)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment.access$withTestCoroutines(ComposeUiTest.android.kt:234)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment$runTest$1$1$1.invoke(ComposeUiTest.android.kt:326)
    at androidx.compose.ui.test.junit4.IdlingStrategy.withStrategy(IdlingStrategy.android.kt:52)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment$runTest$1$1.invoke(ComposeUiTest.android.kt:325)
    at androidx.compose.ui.test.junit4.IdlingResourceRegistry.withRegistry(IdlingResourceRegistry.jvm.kt:157)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment$runTest$1.invoke(ComposeUiTest.android.kt:324)
    at androidx.compose.ui.test.junit4.ComposeRootRegistry.withRegistry(ComposeRootRegistry.android.kt:146)
    at androidx.compose.ui.test.AndroidComposeUiTestEnvironment.runTest(ComposeUiTest.android.kt:323)
    at androidx.compose.ui.test.junit4.AndroidComposeTestRule$apply$1.evaluate(AndroidComposeTestRule.android.kt:271)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.robolectric.RobolectricTestRunner$HelperTestRunner$1.evaluate(RobolectricTestRunner.java:588)
    at org.robolectric.internal.SandboxTestRunner$2.lambda$evaluate$2(SandboxTestRunner.java:290)
    at org.robolectric.internal.bytecode.Sandbox.lambda$runOnMainThread$0(Sandbox.java:101)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
</failure>
  </testcase>
  <testcase name="GIVEN there is an Error state THEN nothing is shown" classname="com.marksandspencer.home.component.banner.ui.TextBannerComponentTest" time="2.724"/>
  <testcase name="GIVEN there is a NotLoaded state THEN nothing is shown" classname="com.marksandspencer.home.component.banner.ui.TextBannerComponentTest" time="0.367"/>
  <testcase name="GIVEN there is a Loaded state THEN loaded text is shown" classname="com.marksandspencer.home.component.banner.ui.TextBannerComponentTest" time="2.632"/>
  <testcase name="GIVEN there is a Loading state THEN loading UI is shown" classname="com.marksandspencer.home.component.banner.ui.TextBannerComponentTest" time="0.714"/>
  <testcase name="GIVEN sut has Loaded state WHEN it is clicked THEN onClicked action with respective banner info is called" classname="com.marksandspencer.home.component.banner.ui.TextBannerComponentTest" time="1.89"/>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>
mikepenz commented 1 month ago

Could it be that the files are existing in 2 locations? Depending on your inclusion filter it might find both? (e.g. in the build directory, and in a differnet directory)?

swwol commented 1 month ago

I thought it might be something like that initially, but it doesn't seem to be - I uploaded the test results to GitHub using the same glob as used for the junit reporter action, and then downloaded that artefact and used grep to search for the failure message ("This test is designed to fail"). There is only one xml file containing that sentence. So I don't think the files are duplicated.

swwol commented 1 month ago

Ah - pretty sure its me doing something silly. (I was downloading the uploaded files and hence creating a copy of them) Apologies!