Flutter test - Test report is showing wrong count of test cases.

[komalkhatri08]

I am running Flutter test command with publish report and code coverage report enabled. While running the task I get same result as in visual studio terminal (EG: 96 passed 1 failed) but in reports on azure pipeline it is showing me 134 test cases and all passed. Attaching snapshots.]

Kindly help me with issue. if anything else is needed please let me know.

[hey24sheep]

Hi, I apologize for the delay. "Test" task was rewritten (as there were same issues with it that you have just reported) and was tested by the community (refer to this issue). No one has reported any issue with it.

I can't help much as I don't know where it could be failing and how can it be failing. As small unit tests to test the "test task" are all passing and community also said it's working fine. I will keep this issue opened up if in case anyone else maybe help here

[komalkhatri08]

<testsuites>
  <testsuite timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="1">
    <testcase name="Some tests failed." classname="Some tests failed." time="0"/>
  </testsuite>
  <testsuite name="auth_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="7">
    <testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/auth_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/auth_model_test.dart" time="9.235"/>
    <testcase name="AuthModelNotifier Tests Set AuthModel" classname="AuthModelNotifier Tests Set AuthModel" time="0.032"/>
    <testcase name="AuthModelNotifier Tests Clear AuthModel" classname="AuthModelNotifier Tests Clear AuthModel" time="0"/>
    <testcase name="AuthModel Tests AuthModel toMap" classname="AuthModel Tests AuthModel toMap" time="0"/>
    <testcase name="AuthModel Tests AuthModel fromMap" classname="AuthModel Tests AuthModel fromMap" time="0.015"/>
    <testcase name="AuthModel Tests AuthModel toJson" classname="AuthModel Tests AuthModel toJson" time="0"/>
    <testcase name="AuthModel Tests AuthModel fromJson" classname="AuthModel Tests AuthModel fromJson" time="0"/>
  </testsuite>
  <testsuite name="device_info_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="2">
    <testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/device_info_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/device_info_model_test.dart" time="0.759"/>
    <testcase name="DeviceInfoModel fromMap should create a DeviceInfoModel from a map" classname="DeviceInfoModel fromMap should create a DeviceInfoModel from a map" time="0"/>
  </testsuite>
  <testsuite name="organization_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="4">
    <testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/organization_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/organization_model_test.dart" time="0.764"/>
    <testcase name="OrganisationModel toJson and fromJson" classname="OrganisationModel toJson and fromJson" time="0.015"/>
    <testcase name="OrganisationModel toMap and fromMap" classname="OrganisationModel toMap and fromMap" time="0"/>
    <testcase name="OrganisationModel toString" classname="OrganisationModel toString" time="0"/>
  </testsuite>
  <testsuite name="role_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="8">
    <testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/role_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/role_model_test.dart" time="0.908"/>
    <testcase name="RoleModel Tests RoleModel should be created with default values" classname="RoleModel Tests RoleModel should be created with default values" time="0.031"/>
    <testcase name="RoleModel Tests RoleModel should be created with values" classname="RoleModel Tests RoleModel should be created with values" time="0"/>
    <testcase name="RoleModel Tests RoleModel should be created from a map" classname="RoleModel Tests RoleModel should be created from a map" time="0"/>
    <testcase name="RoleModel Tests RoleModel should be converted to a map" classname="RoleModel Tests RoleModel should be converted to a map" time="0"/>
    <testcase name="RoleModel Tests RoleModel should be created from JSON" classname="RoleModel Tests RoleModel should be created from JSON" time="0.015"/>
    <testcase name="RoleModel Tests RoleModel should be converted to JSON" classname="RoleModel Tests RoleModel should be converted to JSON" time="0"/>
    <testcase name="RoleModel Tests RoleModel toString() method should return a formatted string" classname="RoleModel Tests RoleModel toString() method should return a formatted string" time="0"/>
  </testsuite>
  <testsuite name="session_data_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="5">
    <testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/session_data_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/session_data_model_test.dart" time="0.734"/>
    <testcase name="SessionDataModel tests Creating SessionDataModel instance" classname="SessionDataModel tests Creating SessionDataModel instance" time="0.03"/>
    <testcase name="SessionDataModel tests CopyWith method" classname="SessionDataModel tests CopyWith method" time="0"/>
    <testcase name="SessionDataModel tests ToMap and fromMap methods" classname="SessionDataModel tests ToMap and fromMap methods" time="0"/>
    <testcase name="SessionDataModel tests ToJson and fromJson methods" classname="SessionDataModel tests ToJson and fromJson methods" time="0"/>
  </testsuite>
  <testsuite name="session_model_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="3">
    <testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/session_model_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/datamodel/session_model_test.dart" time="0.711"/>
    <testcase name="SessionModel Can create SessionModel instance" classname="SessionModel Can create SessionModel instance" time="0.03"/>
    <testcase name="SessionModel Can set and get properties" classname="SessionModel Can set and get properties" time="0"/>
  </testsuite>
  <testsuite name="auth_storage_test.dart" timestamp="2024-02-21T17:33:49.574Z" errors="0" skipped="0" failures="0" tests="5">
    <testcase name="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/providers/auth_storage_test.dart" classname="loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/providers/auth_storage_test.dart" time="0.857"/>
    <testcase name="AuthStorage Tests Save and load token from cache" classname="AuthStorage Tests Save and load token from cache" time="0.03"/>
    <testcase name="AuthStorage Tests Load isLoggedIn from cache" classname="AuthStorage Tests Load isLoggedIn from cache" time="0"/>
    <testcase name="AuthStorage Tests Save and load device token from cache" classname="AuthStorage Tests Save and load device token from cache" time="0.015"/>
    <testcase name="AuthStorage Tests Clear storage" classname="AuthStorage Tests Clear storage" time="0"/>
  </testsuite>

Abve is my junit.xml file ... If I read this file I get extra test case like "loading D:/a/1/s/ISFV2/ericsson_mobile_isf/test/features/authentication/providers/auth_storage_test.dart for every file "_test.dart" file , Also there is a test case for - <testcase name="Some tests failed." I am using version 0 of Flutter Task extension. Is there any other version to update?

[hey24sheep]

"Some tests failed" is generated, that we can't remove because internally it works by reading the STDOUT of the terminal and makes test cases out of that. Looking at the file, I see that the file is generated correctly, each test is having 1 extra case which is the "loading" one, that is what you are saying? But, I think that is correct way of the file, no?

I will keep it open if anyone from the community can help here.

Version for "Flutter Test Task" should be "0.3.5"

[mikoedg]

We are seeing the same as the OP in our pipeline. The published test report, that shows in the Test tab is indicating that all tests have passed but the reality is that some tests are failing.

[hey24sheep]

@mikoedg Hi, I can understand and I appologize. The problem is with the parser and how Azure gives the STDOUT data. I can try and test but I don't have a project in Flutter anymore. I will keep this open if anyone from the community can help.

[tylerbarnesTrexis]

This is still a continued issue, any updates on this?

[hey24sheep]

@tylerbarnesTrexis No update on this, I can't figure out how to fix it. So if anyone can help me fix it that will glad. I don't know if the success rate is extra or failing rate is extra.

[ipuhakka]

Did some minor digging around for this out of curiosity. If tasks/test/index.js createTestCase() is modified to something like this, I think the results might be accurate based on the test cases in sample project.

function createTestCase(suite, output, globalFailure) {
    var testRunRegex = /\s*\d\d:\d\d (\+\d+)?(\s+\-\d+)?:\s*(.*)/;
    var match = testRunRegex.exec(output);
    if (!match || match.length < 4) {
        return suite;
    }
    var tSplits = match[3].split(': ');

    if (tSplits.length === 1) {
        // Log printed starting a new test suite - No test case handled here
        return suite;
    }

    // Sanitise test case name from other information
    let title = tSplits[tSplits.length - 1];
    title = title.replace(' [E]', '');
    const currentCase = suite.cases.find(testCase => testCase.title === title);

    if (!currentCase) {
        // Suite has to be marked as succeeded first, marking as failed can be done when there is a new row with [E] at the end
        suite.cases.push({
            title: title.trim(),
            isSuccess: true,
            started: new Date(),
            ended: new Date(),
        });
        suite.succeeded += 1;
        return suite;
    }

    if (match[3].endsWith(' [E]')) {
        currentCase.isSuccess = false;
        suite.failed += 1;
        suite.succeeded -= 1;
    }

    return suite;
}

It looked to me like there were two test case results for cases which fail. That being said, I don't know anything about the test runner and parsing these results (this was just a small naive test).

Edit: To clarify, each individual case showed as succeeded, each error row of the output was its own test case as well as the "loading.." text. Test cases before the change:

{
    "isSuccess": false,
    "suites": [
        {
            "title": "1_api_test.dart",
            "isSuccess": false,
            "succeeded": 6,
            "failed": 0,
            "cases": [
                {
                    "title": "loading C:/Git/azure-flutter-tasks/sample_project/test/1_api_test.dart",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:13.566Z",
                    "ended": "2024-09-25T10:35:16.075Z"
                },
                {
                    "title": "1_api_t1",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:16.075Z",
                    "ended": "2024-09-25T10:35:16.098Z"
                },
                {
                    "title": "1_api_t2",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:16.098Z",
                    "ended": "2024-09-25T10:35:16.102Z"
                },
                {
                    "title": "1_api_t3",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:16.102Z",
                    "ended": "2024-09-25T10:35:16.104Z"
                },
                {
                    "title": "1_api_fail_t4",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:16.104Z",
                    "ended": "2024-09-25T10:35:16.167Z"
                },
                {
                    "title": "1_api_fail_t4 [E]",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:16.167Z",
                    "ended": "2024-09-25T10:35:16.167Z"
                }
            ]
        },
        {
            "title": "2_api_test.dart",
            "isSuccess": false,
            "succeeded": 1,
            "failed": 0,
            "cases": [
                {
                    "title": "2_api_t1",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:15.964Z",
                    "ended": "2024-09-25T10:35:15.964Z"
                }
            ]
        },
        {
            "isSuccess": false,
            "succeeded": 1,
            "failed": 0,
            "cases": [
                {
                    "title": "Some tests failed.",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:17.881Z",
                    "ended": "2024-09-25T10:35:17.881Z"
                }
            ]
        },
        {
            "title": "3_services_test.dart",
            "isSuccess": false,
            "succeeded": 2,
            "failed": 0,
            "cases": [
                {
                    "title": "3_service_t1",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:16.173Z",
                    "ended": "2024-09-25T10:35:16.212Z"
                },
                {
                    "title": "3_service_t2",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:16.212Z",
                    "ended": "2024-09-25T10:35:16.212Z"
                }
            ]
        },
        {
            "title": "4_widget_test.dart",
            "isSuccess": false,
            "succeeded": 3,
            "failed": 0,
            "cases": [
                {
                    "title": "4_widget_t1",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:16.430Z",
                    "ended": "2024-09-25T10:35:16.981Z"
                },
                {
                    "title": "4_widget_t2",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:16.981Z",
                    "ended": "2024-09-25T10:35:17.030Z"
                },
                {
                    "title": "4_widget_t3",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:17.030Z",
                    "ended": "2024-09-25T10:35:17.030Z"
                }
            ]
        },
        {
            "title": "5_widget_test.dart",
            "isSuccess": false,
            "succeeded": 1,
            "failed": 0,
            "cases": [
                {
                    "title": "5_widget_t1",
                    "isSuccess": true,
                    "started": "2024-09-25T10:35:17.320Z",
                    "ended": "2024-09-25T10:35:17.320Z"
                }
            ]
        }
    ]
}

And with the change:

{
    "isSuccess": false,
    "suites": [
        {
            "title": "1_api_test.dart",
            "isSuccess": false,
            "succeeded": 3,
            "failed": 1,
            "cases": [
                {
                    "title": "1_api_t1",
                    "isSuccess": true,
                    "started": "2024-09-25T16:33:03.686Z",
                    "ended": "2024-09-25T16:33:03.686Z"
                },
                {
                    "title": "1_api_t2",
                    "isSuccess": true,
                    "started": "2024-09-25T16:33:03.741Z",
                    "ended": "2024-09-25T16:33:03.741Z"
                },
                {
                    "title": "1_api_t3",
                    "isSuccess": true,
                    "started": "2024-09-25T16:33:03.748Z",
                    "ended": "2024-09-25T16:33:03.748Z"
                },
                {
                    "title": "1_api_fail_t4",
                    "isSuccess": false,
                    "started": "2024-09-25T16:33:03.756Z",
                    "ended": "2024-09-25T16:33:03.756Z"
                }
            ]
        },
        {
            "title": "2_api_test.dart",
            "isSuccess": false,
            "succeeded": 1,
            "failed": 0,
            "cases": [
                {
                    "title": "2_api_t1",
                    "isSuccess": true,
                    "started": "2024-09-25T16:33:04.108Z",
                    "ended": "2024-09-25T16:33:04.108Z"
                }
            ]
        },
        {
            "title": "3_services_test.dart",
            "isSuccess": false,
            "succeeded": 2,
            "failed": 0,
            "cases": [
                {
                    "title": "3_service_t1",
                    "isSuccess": true,
                    "started": "2024-09-25T16:33:04.487Z",
                    "ended": "2024-09-25T16:33:04.487Z"
                },
                {
                    "title": "3_service_t2",
                    "isSuccess": true,
                    "started": "2024-09-25T16:33:04.547Z",
                    "ended": "2024-09-25T16:33:04.547Z"
                }
            ]
        },
        {
            "title": "4_widget_test.dart",
            "isSuccess": false,
            "succeeded": 1,
            "failed": 0,
            "cases": [
                {
                    "title": "4_widget_t1",
                    "isSuccess": true,
                    "started": "2024-09-25T16:33:05.014Z",
                    "ended": "2024-09-25T16:33:05.014Z"
                }
            ]
        },
        {
            "title": "5_widget_test.dart",
            "isSuccess": false,
            "succeeded": 1,
            "failed": 0,
            "cases": [
                {
                    "title": "5_widget_t1",
                    "isSuccess": true,
                    "started": "2024-09-25T16:33:06.651Z",
                    "ended": "2024-09-25T16:33:06.651Z"
                }
            ]
        }
    ]
}

[hey24sheep]

Hi all, @ipuhakka thanks for the code I incorporated it and release a version 0.4.4. Please check and let me know if it fixed the issue. @tylerbarnesTrexis @mikoedg @komalkhatri08

[hey24sheep]

Hi, can anyone update me so I can close this issue if it's fixed. Thank you