"All tests passed!" reported as failing test

[aspernol]

Hi, I'm just setting up an azure pipeline for a flutter application and using azure flutter tasks. When running unit tests I have the problem that the final "All tests passed!" message is shown in the report as failing test. Currently I have only setup one dummy test in my project just to get test execution and code coverage running on azure devops. Here is my configuration:

    - task: FlutterTest@0
      displayName: "Run unit tests"
      continueOnError: true
      inputs:
        generateCodeCoverageReport: true
        projectDirectory: '$(Build.SourcesDirectory)/configuration_app'
        extraArgs: '--branch-coverage'

Here is the result log:

Starting: Run unit tests
==============================================================================
Task         : Flutter Test Task
Description  : Executes all tests for a Flutter project.
Version      : 0.3.3
Author       : Hey24sheep
Help         : For more information, take a look at the Flutter [documentation](https://flutter.io/)
==============================================================================
C:\Windows\system32\cmd.exe /D /S /C "C:\hostedtoolcache\windows\Flutter\3.13.1\windows\flutter\bin\flutter.bat test --pub --coverage --branch-coverage"
00:00 +0: Some non meaningful test
00:01 +1: All tests passed!
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Async Command Start: Publish test results
Publishing test results to test run '72'.
TestResults To Publish 2, Test run id:72
Test results publishing 2, remaining: 0. Test run id: 72
Published Test Run : https://dev.azure.com/traxon/light-x/_TestManagement/Runs?runId=72&_a=runCharts
Flaky failed test results are opted out of pass percentage
Async Command End: Publish test results
Finishing: Run unit tests

As you can see in the logs there is only one test executed but "TestResults To Publish" is 2. I tried to find an option to suppress the "All tests passed!" message but couldn't find anything.

[hey24sheep]

Hi, this issue was created while fixing a previous issue #83. It's a complete rewrite of the whole testing part so it has some bugs and inconsistency.

Could you provide the generate publish results?

I can't reproduce it on my local machine.

[aspernol]

Hi, you mean the junit.xml? Or is there some other result. I fetched this one from the server.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<testsuites>
  <testsuite timestamp="2023-08-28T02:30:18.604Z" errors="0" skipped="0" failures="1" tests="2">
    <testcase name="Some non meaningful test" classname="Some non meaningful test" time="0"/>
    <testcase name="All tests passed!" classname="All tests passed!" time="0">
      <failure type="FlutterError"/>
    </testcase>
  </testsuite>
</testsuites>

[Okamael]

I'm having the same problem, I tested it with the build on mac and windows, on windows it ends up identifying 4 tests when in fact I only have 3 and the 3 are passing

On mac there is duplication for the same tests

[Iamkosgei]

This is happening to me as well, all tests pass locally.

[hey24sheep]

@Iamkosgei @Okamael the issue is I am unable to reproduce it on my local. Reason being first I am little too busy with work and second I don't have enough real test data to mock the results.

The way to read and create testsuites is actually based on reading the data from STDIN string. There is no other way. So, maybe in that case, it might be that it's getting double log string and creating it as false fail.

Current scenario is test report consists of "False Failures" other than that everything else seems to work with the "Test Task"

I'll re-work on this later on once I get a little bit extra time.

[Iamkosgei]

@hey24sheep thank you. In case anyone else is facing this issue, you can use bash as we wait for the fix.

    - task: Bash@3
      displayName: 'Run tests'
      inputs:
        targetType: 'inline'
        script: |
          export PATH="$PATH":"$HOME/.pub-cache/bin"
          flutter pub global activate junitreport
          flutter pub get
          flutter test --machine | tojunit -o junit.xml
        workingDirectory: '$(Build.SourcesDirectory)'

    - task: PublishTestResults@2
      displayName: 'Publish Test Results'
      condition: succeeded()
      inputs:
        testResultsFormat: 'JUnit'
        testResultsFiles: '**/junit.xml'
        failTaskOnFailedTests: true

[hey24sheep]

@Iamkosgei @Okamael @aspernol I have updated the extension, please validate and let me know if you are still facing this issue.

[hey24sheep]

@Iamkosgei @Okamael @aspernol If you guys could let me know if the fix works, I will close this issue.

[aspernol]

Hi, Sorry I must have missed it. I'm away for a few days but I will give you feedback end of next week, when I'm Back

Heyramb Narayan Goyal @.***> schrieb am Fr., 27. Okt. 2023, 07:32:

@Iamkosgei https://github.com/Iamkosgei @Okamael https://github.com/Okamael @aspernol https://github.com/aspernol If you guys could let me know if the fix works, I will close this issue.

— Reply to this email directly, view it on GitHub https://github.com/hey24sheep/azure-flutter-tasks/issues/101#issuecomment-1782321271, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALC54EWL7ZXFBQNPJBSOJPLYBNBQPAVCNFSM6AAAAAA37FWNXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBSGMZDCMRXGE . You are receiving this because you were mentioned.Message ID: @.***>

[Iamkosgei]

@hey24sheep sorry I missed this. Just tested and works fine now. Thank you.

[Okamael]

@Iamkosgei @Okamael @aspernol I have updated the extension, please validate and let me know if you are still facing this issue.

works fine now! thanks

[hey24sheep]

@Iamkosgei @Okamael thanks for the feedback.

[aspernol]

@hey24sheep Sorry again for the late response. Works fine now. Many thanks.