Multiple testcase IDs report incorrect status to TestRail

[spencerlavallesonos]

When I pass multiple CaseIDs via the it block, the result posted to testrail is an AND of all test case results (E.g. all true results in TestRail report showing all true, any failed = all failed, all failed = all failed).

Image showing test output + testrail's test case test results.

Spec used: (Note: the forEach is just looping through en-us, es-us, en-ca, and fr-ca, and Command is just a helper class)

import * as EnvironmentHelper from '../src/base/EnvironmentHelper'
const { addTestRailComment } = require('wdio-wdiov5testrail-reporter').default

describe('Spencer TestRail Test', async function () {
  EnvironmentHelper.testLocales.forEach(function (locale) {
    it(`C33835691, C33904020 Fake TestRail Test @\t ${locale}`, async function () {
      Command.log(`Running Test :${locale}`)
      addTestRailComment(`Locale:\t${locale}`)
      if (locale === 'en-ca') {
        expect(true).to.be.equal(false)
      }
    })
  })
})

[billylam]

To check, is the issue that the results and comments section should report 3 passes and 1 failure? The overall result is intended to fail if any fail.

[billylam]

Closed while awaiting info.

[spencerlavallesonos]

@billylam That would be my assumption that not all 4 would be reported as failures as according to WDIO not all 4 of those tests failed, only the single locale. IMO it seems that reporting all 4 as failed is reporting the test status incorrectly.

Please let me know if you need any other info :smile:

[billylam]

Unfortunately, I no longer use webdriverio or have a TestRail instance so won't be able to test a fix.

If you use v8+, it looks like there is now an official package.

If you are not, I believe this can be fixed by changing the logic in the block at https://github.com/billylam/wdio-wdiov5testrail-reporter/blob/master/src/util.js#L127. I THINK the fix would be to change to something like

    if (testCaseIds.length !== new Set(testCaseIds).size) {
      const failures = results
        .filter((result) => result.status_id !== 1);
      const successes = results.filter((result => result.status_id === 1))
        .map((result) => result.case_id);
      results = successes.concat(failures);
    }

If you find this helps, let me know so I can create a commit or feel free to submit a pull request yourself.

[spencerlavallesonos]

Hey Billy thanks so much, when I have some free time in a few weeks to pick this back up I'll investigate and give this a go. Thank you!