search

EnricoMi / publish-unit-test-result-action

GitHub Action to publish unit test results on GitHub
Apache License 2.0
605 stars 179 forks source link

Include test output in the check #354

Closed mikz closed 1 year ago

mikz commented 2 years ago

Unity NUnit test runner outputs what was written to the console when the test was running and that is extremely valuable information

Here is a stripped down example of one test run.

<?xml version="1.0" encoding="utf-8"?>
<test-run id="2" testcasecount="352" result="Failed(Child)" total="352" passed="349" failed="3" inconclusive="0" skipped="0" asserts="0" engine-version="3.5.0.0" clr-version="4.0.30319.42000" start-time="2022-09-20 21:16:48Z" end-time="2022-09-20 21:31:37Z" duration="888.5251836">
    <test-suite type="TestSuite" id="1000" name="workspace" fullname="workspace" runstate="Runnable" testcasecount="3688" result="Failed" site="Child" start-time="2022-09-20 21:16:48Z" end-time="2022-09-20 21:31:37Z" duration="888.525184" total="352" passed="349" failed="3" inconclusive="0" skipped="0" asserts="0">
        <properties>
            <property name="isRoot" value="True" />
        </properties>
        <failure>
            <message><![CDATA[One or more child tests had errors]]></message>
        </failure>
        <test-suite type="Assembly" id="1361" name="MafiaPigs.EditorTests.dll" fullname="/github/workspace/Library/ScriptAssemblies/MafiaPigs.EditorTests.dll" runstate="Runnable" testcasecount="352" result="Failed" site="Child" start-time="2022-09-20 21:16:48Z" end-time="2022-09-20 21:31:37Z" duration="888.385199" total="352" passed="349" failed="3" inconclusive="0" skipped="0" asserts="0">
            <properties>
                <property name="_PID" value="1140" />
                <property name="_APPDOMAIN" value="Unity Child Domain" />
                <property name="platform" value="EditMode" />
                <property name="isAssembly" value="True" />
                <property name="EditorOnly" value="True" />
                <property name="RequiresPlayMode" value="False" />
            </properties>
            <failure>
                <message><![CDATA[One or more child tests had errors]]></message>
            </failure>
            <test-suite type="TestSuite" id="1362" name="MP" fullname="MP" runstate="Runnable" testcasecount="352" result="Failed" site="Child" start-time="2022-09-20 21:16:48Z" end-time="2022-09-20 21:31:37Z" duration="888.339072" total="352" passed="349" failed="3" inconclusive="0" skipped="0" asserts="0">
                <properties />
                <failure>
                    <message><![CDATA[One or more child tests had errors]]></message>
                </failure>
                <test-suite type="TestSuite" id="1363" name="Tests" fullname="MP.Tests" runstate="Runnable" testcasecount="352" result="Failed" site="Child" start-time="2022-09-20 21:16:48Z" end-time="2022-09-20 21:31:37Z" duration="888.307736" total="352" passed="349" failed="3" inconclusive="0" skipped="0" asserts="0">
                    <properties />
                    <failure>
                        <message><![CDATA[One or more child tests had errors]]></message>
                    </failure>
                    <test-suite type="TestFixture" id="1003" name="AssetValidatorTest" fullname="MP.Tests.AssetValidatorTest" classname="MP.Tests.AssetValidatorTest" runstate="Runnable" testcasecount="351" result="Failed" site="Child" start-time="2022-09-20 21:16:48Z" end-time="2022-09-20 21:31:36Z" duration="887.960312" total="351" passed="348" failed="3" inconclusive="0" skipped="0" asserts="0">
                        <properties />
                        <failure>
                            <message><![CDATA[One or more child tests had errors]]></message>
                        </failure>
                        <output/>
                        <test-suite type="ParameterizedMethod" id="1062" name="ValidateSceneContainer" fullname="MP.Tests.AssetValidatorTest.ValidateSceneContainer" classname="MP.Tests.AssetValidatorTest" runstate="Runnable" testcasecount="56" result="Failed" site="Child" start-time="2022-09-20 21:24:21Z" end-time="2022-09-20 21:28:24Z" duration="242.562296" total="56" passed="53" failed="3" inconclusive="0" skipped="0" asserts="0">
                            <properties />
                            <failure>
                                <message><![CDATA[One or more child tests had errors]]></message>
                            </failure>
                            <test-case id="1017" name="ValidateSceneContainer(&quot;Assets/Scenes/Grid/GridTest.unity&quot;)" fullname="MP.Tests.AssetValidatorTest.ValidateSceneContainer(&quot;Assets/Scenes/Grid/GridTest.unity&quot;)" methodname="ValidateSceneContainer" classname="MP.Tests.AssetValidatorTest" runstate="Runnable" seed="78278819" result="Failed" label="Error" start-time="2022-09-20 21:25:49Z" end-time="2022-09-20 21:25:51Z" duration="2.117365" asserts="0">
                                <properties />
                                <failure>
                                    <message><![CDATA[Zenject.ZenjectException : Zenject Validation Failed!  See errors below for details.]]></message>
                                    <stack-trace><![CDATA[  at Zenject.Internal.ZenUnityEditorUtil.ValidateCurrentSceneSetup () [0x0009c] in /github/workspace/Assets/ThirdParty/Zenject/Source/Editor/ZenUnityEditorUtil.cs:82
  at MP.Tests.AssetValidatorTest.ValidateSceneContainer (System.String scenePath) [0x00009] in /github/workspace/Assets/Tests/EditorMode/AssetValidatorTest.cs:58
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <b67d2f60bf2548a58dc569b37fe71c3d>:0 ]]></stack-trace>
                                </failure>
                                <output><![CDATA[AssertionException: Could not find a tilemap tagged with LevelBounds.
Assertion failure. Value was Null
Expected: Value was not Null
UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) (at /home/bokken/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertBase.cs:29)
UnityEngine.Assertions.Assert.IsNotNull (UnityEngine.Object value, System.String message) (at /home/bokken/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertNull.cs:58)
UnityEngine.Assertions.Assert.IsNotNull[T] (T value, System.String message) (at /home/bokken/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertNull.cs:46)
MP.Gameplay.Level.LevelInstaller.InstallBindings () (at Assets/Scripts/Gameplay/Level/LevelInstaller.cs:30)
Zenject.CompositeMonoInstaller.InstallBindings () (at Assets/ThirdParty/Zenject/Source/Install/CompositeMonoInstaller.cs:25)
Zenject.Context.InstallInstallers (System.Collections.Generic.List`1[T] normalInstallers, System.Collections.Generic.List`1[T] normalInstallerTypes, System.Collections.Generic.List`1[T] scriptableObjectInstallers, System.Collections.Generic.List`1[T] installers, System.Collections.Generic.List`1[T] installerPrefabs) (at Assets/ThirdParty/Zenject/Source/Install/Contexts/Context.cs:218)
Zenject.Context.InstallInstallers () (at Assets/ThirdParty/Zenject/Source/Install/Contexts/Context.cs:139)
Zenject.SceneContext.InstallBindings (System.Collections.Generic.List`1[T] injectableMonoBehaviours) (at Assets/ThirdParty/Zenject/Source/Install/Contexts/SceneContext.cs:346)
Zenject.SceneContext.Install () (at Assets/ThirdParty/Zenject/Source/Install/Contexts/SceneContext.cs:265)
Zenject.SceneContext.Validate () (at Assets/ThirdParty/Zenject/Source/Install/Contexts/SceneContext.cs:121)
Zenject.Internal.ZenUnityEditorUtil.ValidateCurrentSceneSetup () (at Assets/ThirdParty/Zenject/Source/Editor/ZenUnityEditorUtil.cs:67)
UnityEngine.Debug:LogException(Exception)
ModestTree.Log:ErrorException(Exception) (at Assets/ThirdParty/Zenject/Source/Internal/Log.cs:60)
Zenject.Internal.ZenUnityEditorUtil:ValidateCurrentSceneSetup() (at Assets/ThirdParty/Zenject/Source/Editor/ZenUnityEditorUtil.cs:72)
MP.Tests.AssetValidatorTest:ValidateSceneContainer(String) (at Assets/Tests/EditorMode/AssetValidatorTest.cs:58)
System.Reflection.MethodBase:Invoke(Object, Object[])
NUnit.Framework.Internal.Reflect:InvokeMethod(MethodInfo, Object, Object[])
NUnit.Framework.Internal.MethodWrapper:Invoke(Object, Object[])
NUnit.Framework.Internal.Commands.TestMethodCommand:RunNonAsyncTestMethod(ITestExecutionContext)
NUnit.Framework.Internal.Commands.TestMethodCommand:RunTestMethod(ITestExecutionContext)
NUnit.Framework.Internal.Commands.TestMethodCommand:Execute(ITestExecutionContext)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions() (at /home/bokken/buildslave/unity/build/Editor/Mono/EditorApplication.cs:359)

]]></output>
                            </test-case>
                        </test-suite>
                    </test-suite>
                </test-suite>
            </test-suite>
        </test-suite>
    </test-suite>
</test-run>

It would be awesome if it would be possible to have the contents of the <output> tag in the GitHub check. Right now there is just stack trace, without the error message. Which makes it very hard to figure out why it failed and you have to download the xml file and dig by hand anyway.

Screenshot 2022-09-21 at 17 25 15
mikz commented 2 years ago

I'm happy to actually implement it if you could give me some pointers where. I tried to dig but wasn't really sure how to approach it. Ideally it should be a configurable flag.

EnricoMi commented 2 years ago

I agree, the output tag contains valuable information.

I suspect this would require to add something like

<xsl:value-of select="../output"/>

after https://github.com/EnricoMi/publish-unit-test-result-action/blob/master/python/publish/xslt/nunit3-to-junit.xslt#L119

<xsl:value-of select="./stack-trace"/>

No configuration option needed, as this should always be included if <output> exists.

EnricoMi commented 2 years ago

On a side note: the message should be included. I would consider that a separate bug.

mikz commented 2 years ago

👍 I'll do and test it on my project.

mikz commented 2 years ago

I've tried in https://github.com/timewarpinc/publish-unit-test-result-action/blob/0c67e0e98bc1eb2db461189768537a2b30eb7928/python/publish/xslt/nunit3-to-junit.xslt and executed the transform locally on the nunit test output.

Here is relevant part of the transformed xml:

<testcase name="OdinValidator" classname="" status="Failed" time="417.005347">
              <failure message="UnityEngine.Assertions.AssertionException : Assertion failure. Values are not equal.&#10;Expected: 19 == 0">UnityEngine.Assertions.AssertionException : Assertion failure. Values are not equal.
Expected: 19 == 0  at UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) [0x0003c] in /home/bokken/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertBase.cs:29
  at UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message, System.Collections.Generic.IEqualityComparer`1[T] comparer) [0x0004d] in /home/bokken/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertGeneric.cs:31
  at UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message) [0x00001] in /home/bokken/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertGeneric.cs:19
  at UnityEngine.Assertions.Assert.AreEqual (System.Int32 expected, System.Int32 actual) [0x0000c] in /home/bokken/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertPrimitiveTypes.cs:176
  at MP.Tests.AssetValidatorTest.OdinValidator () [0x000a7] in /github/workspace/Assets/Tests/EditorMode/AssetValidatorTest.cs:56
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&amp;)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in &lt;b67d2f60bf2548a58dc569b37fe71c3d&gt;:0 Added AStarManager (Zenject.CompositeMonoInstaller) to Managers (Zenject.SceneContext)
Error: AStar Pathfinder grid does not contain level bounds.
Added AStarManager (Zenject.CompositeMonoInstaller) to SceneContext (Zenject.SceneContext)
Material Sprite-Lit-Default (UnityEngine.Material) not have GLOW_ON keyword
Completed validating 720073 values in 416.8882977 seconds.
</failure>
              <system-out>Error: AStar Pathfinder grid does not contain level bounds.
Added AStarManager (Zenject.CompositeMonoInstaller) to SceneContext (Zenject.SceneContext)
Material Sprite-Lit-Default (UnityEngine.Material) not have GLOW_ON keyword
Completed validating 720073 values in 416.8882977 seconds.
</system-out>
            </testcase>

So the transformation works. But the action is producing the same github check as before. And I've pinned it to the exact commit, so I don't get why it does not work. Likely because it is using different transform.

But that got me thinking. It already transforms the system-out. Isn't it better to actually use system-out from python and add it to the check? That would be a bit cleaner IMO.

mikz commented 2 years ago

Ok, it does not work from a fork because the action is set up to use your published docker image.

EnricoMi commented 1 year ago

You can create a PR in your fork targeting your fork master and make .github/workflows/test-results.yml in your master use your the action from your branch.

EnricoMi commented 1 year ago

Run the action with

with:
  root_log_level: DEBUG
  log_level: DEBUG

then you might be able to see the payload that is sent to GitHub API.

EnricoMi commented 1 year ago

I have added stdout and stderror to the annotations output.

I have created a fix, please test the fix as follows (includes fix for #355):

uses: EnricoMi/publish-unit-test-result-action/composite@branch-stdout-and-stderr
mikz commented 1 year ago

You can create a PR in your fork targeting your fork master and make .github/workflows/test-results.yml in your master use your the action from your branch.

I though so too, but the action is configured to run a docker image and not use the repo at all. So none of the code is actually evaluated. Only the composite action works.

https://github.com/EnricoMi/publish-unit-test-result-action/blob/713caf1dd6f1c273144546ed2d79ca24a01f4623/action.yml#L111

mikz commented 1 year ago

I have added stdout and stderror to the annotations output.

I have created a fix, please test the fix as follows (includes fix for #355):

uses: EnricoMi/publish-unit-test-result-action/composite@branch-stdout-and-stderr

Thanks! Sorry for the delay. I'll check it and report. I'll check 2.1.0.