A clear and concise description of what the bug is.
Hi! Thanks for this action!
I have a Flutter app that has thousands of tests. In CI, we create a matrix to run these tests on 15 parallel nodes. Each of these notes creates an individual json test report output that we upload as an artifact. When each node is done running its test, we combine all of these files into one file so that we can upload it to the test-reporter action and post the run's results to the pr.
The problem is that since we're combining multiple test files, we have repeating testId's in the test report. ie, each parallel node has a testId:1 in its report. The way that the test reporter calculates the results doesn't account for this possibility. This means that if, say, the 3rd test 1 fails, but the other Test 1's succeeded, the test reporter doesn't mark test 1 as a failure and reports that all tests passed.
It also means, by the same logic, that the test reporter is miscalculating the total number of tests run. If I have 15 test 1's, it only counts 1, so the total test count is 1 rather than 15.
To Reproduce
Steps to reproduce the behavior:
I've created a very simple repro app that can be found here
I then combined these test results into test_report_combined.json to mimic what we do in our larger repo.
I opened a pr that uplaods test_report_combined.json to test-reporter and I would expect there to be 2 total tests reported, 1 success, one failure. Instead, it reports 1 test run, 1 success
Expected behavior
Either:
ability to upload one test result file that combines prior test results and reports the expected total number of tests and failures
or:
ability to upload multiple test result files that are combined into one test output that reports the expected total number of tests and failures
Describe the bug
A clear and concise description of what the bug is.
Hi! Thanks for this action!
I have a Flutter app that has thousands of tests. In CI, we create a matrix to run these tests on 15 parallel nodes. Each of these notes creates an individual json test report output that we upload as an artifact. When each node is done running its test, we combine all of these files into one file so that we can upload it to the
test-reporteraction and post the run's results to the pr.The problem is that since we're combining multiple test files, we have repeating
testId's in the test report. ie, each parallel node has atestId:1in its report. The way that the test reporter calculates the results doesn't account for this possibility. This means that if, say, the 3rd test 1 fails, but the other Test 1's succeeded, the test reporter doesn't mark test 1 as a failure and reports that all tests passed.It also means, by the same logic, that the test reporter is miscalculating the total number of tests run. If I have 15 test 1's, it only counts 1, so the total test count is 1 rather than 15.
To Reproduce
Steps to reproduce the behavior:
I've created a very simple repro app that can be found here
This is just the standard Flutter app.
I ran the built in test twice, once with a succes, once with a failure and exported the output to test_report_1.json and test_report_2.json
I then combined these test results into test_report_combined.json to mimic what we do in our larger repo.
I opened a pr that uplaods
test_report_combined.jsontotest-reporterand I would expect there to be 2 total tests reported, 1 success, one failure. Instead, it reports 1 test run, 1 successExpected behavior
Either: