search

NasAmin / trx-parser

This repository contains a Typescript GitHub action for parsing trx files and creating status check for each trx file based on its test outcome
MIT License
15 stars 16 forks source link

Reporting failing tests despite all tests running green #311

Open acn-sbuad opened 3 weeks ago

acn-sbuad commented 3 weeks ago

We are using this action in our workflows to better present the test reports after running a workflow. However, lately we have experienced the trx-parser action failing and reporting on failed tests when it is logged that all test cases have passed.

The worfklow

log from the task running the tests

Test run for /home/runner/work/altinn-notifications/altinn-notifications/test/Altinn.Notifications.Tests/bin/Release/net8.0/Altinn.Notifications.Tests.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: /home/runner/work/altinn-notifications/altinn-notifications/TestResults/_fv-az1052-128_2024-08-19_12_52_21.trx

Passed!  - Failed:     0, Passed:   269, Skipped:     0, Total:   269, Duration: 644 ms - Altinn.Notifications.Tests.dll (net8.0)
Test run for /home/runner/work/altinn-notifications/altinn-notifications/test/Altinn.Notifications.IntegrationTests/bin/Release/net8.0/Altinn.Notifications.IntegrationTests.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
%4|1724071950.202|GETPID|rdkafka#producer-1| [thrd:main]: Failed to acquire idempotence PID from broker localhost:9092/1: Broker: Coordinator load in progress: retrying
Results File: /home/runner/work/altinn-notifications/altinn-notifications/TestResults/_fv-az[105](https://github.com/Altinn/altinn-notifications/actions/runs/10453675204/job/28944613679#step:11:106)2-128_2024-08-19_12_52_32.trx

Passed!  - Failed:     0, Passed:   112, Skipped:     0, Total:   112, Duration: 2 m 47 s - Altinn.Notifications.IntegrationTests.dll (net8.0)

Attachments:
  /home/runner/work/altinn-notifications/altinn-notifications/TestResults/07f1e20a-abd9-4e89-b7e3-c4e6f22c186e/coverage.opencover.xml
  /home/runner/work/altinn-notifications/altinn-notifications/TestResults/f5447407-a6c1-4ffd-b1ca-3c0cf28c63f0/coverage.opencover.xml

log from the trx-parser task: 

Run NasAmin/trx-parser@v0.6.0
Finding Trx files in: /home/runner/work/altinn-notifications/altinn-notifications/TestResults
Files count: 6
Processing 2 trx files
Transforming file /home/runner/work/altinn-notifications/altinn-notifications/TestResults/_fv-az[10](https://github.com/Altinn/altinn-notifications/actions/runs/10453675204/job/28944613679#step:13:11)52-128_2024-08-19_12_52_21.trx
Transforming file /home/runner/work/altinn-notifications/altinn-notifications/TestResults/_fv-az1052-[12](https://github.com/Altinn/altinn-notifications/actions/runs/10453675204/job/28944613679#step:13:13)8_2024-08-19_12_52_32.trx
Checking for failing tests
Creating PR check for ALTINN NOTIFICATIONS.TESTS
Check time is: Mon, 19 Aug 2024 12:55:57 GMT
Created check: altinn.notifications.tests-check with response status 201
Creating PR check for ALTINN NOTIFICATIONS.INTEGRATIONTESTS
Check time is: Mon, 19 Aug 2024 12:55:58 GMT
Created check: altinn.notifications.integrationtests-check with response status 201
Error: At least one failing test was found
acn-sbuad commented 3 weeks ago

Added step to download the test report, and error is written to console output. Assuming this is the reason for the report step also reporting failure despite all tests passing. Is this by design or a bug?

The test result file as xls, converted from .trx

NasAmin commented 3 weeks ago

Thanks for the report. I'll look into this.

NasAmin commented 1 week ago

@acn-sbuad is this still an issue?