stateCounts returning incorrect test results for iOS test runs

[sherryleneg]

Hi,

I have just done the upgrade to the Slack reporter for webdriver v8. As I was testing, I noticed that the iOS Slack notification contains some incorrect test result information. Whenever a feature file fails, it is being marked as passed and hence the notification doesn't come through in our Slack channel. I am overriding the createResultPayload() function to customise the Slack message so that it posts only on failure, however even with the default behaviour (i.e. no customisation) I see that the test results are incorrect for iOS. On the otherhand, Android works as expected.

Code snippet:

`config.reporters.push([
    SlackReporter,
    {
      createFailedTestPayload: function () {
        const payload = {
          //override default function and don't notify in Slack
        }
        return payload
      },
      createResultPayload(runnerStats, stateCounts) {
        const platform = 
        runnerStats.capabilities['platformName']

        const deviceIOS = 
          runnerStats.capabilities['deviceName']

        const deviceAndroid = 
          runnerStats.capabilities['device']

        const buildName = 
          runnerStats.capabilities['buildName']

        //If spec file passes, do not post on Slack
        if (stateCounts.failed === 0) {
         return
        }

        let device = platform === 'Android' ? deviceAndroid : deviceIOS

        const payload = {
          attachments: [
            {
              color: '#FF0000',
              text: `*Results*: <${process.env.CI_JOB_URL}> | *Build*: ${buildName}\n*Platform*: ${platform} | *Device*: ${device}\n*Passed*: ${stateCounts.passed} | *Failed*: ${stateCounts.failed}\n*Pipeline triggered by*: ${process.env.GITLAB_USER_NAME}\n*Branch*: ${process.env.CI_COMMIT_REF_NAME}`,
              ts: Date.now().toString(),
            },
          ],
          blocks: [
            {
              text: {
                emoji: true,
                text: `${runnerStats.specs}`,
                type: 'plain_text',
              },
              type: 'section',
            },
          ],
        }

        return payload
      },
      createStartPayload: function () {
        const payload = {
          //override default function and don't notify in Slack
        }
        return payload
      },

      notifyTestFinishMessage: true,

      slackOptions: {
        slackIconUrl: 'https://webdriver.io/img/webdriverio.png',
        slackName: 'WebdriverIO Reporter',
        type: 'webhook',
        webhook: process.env.SLACK_TEST_WEBHOOK || '',
      },

      title: '',
    },
  ])`

Some context: We use Browserstack service together with Appium to run tests cross platform.

Can you help with this please?

[morooLee]

@sherryleneg Yes. I'll check it out.

[morooLee]

@sherryleneg Could you please share the run log on iOS?

[sherryleneg]

@morooLee This is the job log from the GitLab test run run-log-ios-gitlab.txt

[morooLee]

@sherryleneg Checking the logs, if the test fails, it requests the slack webhook, but the data is either missing or undefined . So, it is being responded with 400 Bad request. The problem seems to be happening on iOS when forking between iOS and Android.

In iOS, you need to check by adding the value where the payload is defined to the log.

[Data is Empty] : DATA {}

[0-10] 2023-07-25T09:56:31.343Z INFO @moroo/wdio-slack-reporter: POST Slack Request webhook:send [0-10] 2023-07-25T09:56:31.344Z DEBUG @moroo/wdio-slack-reporter: DATA {} [0-10] 2023-07-25T09:56:31.655Z ERROR @moroo/wdio-slack-reporter: Error: An HTTP protocol error occurred: statusCode = 400 [0-10] at httpErrorWithOriginal (/builds/virginmediaO2/digital/hhh/journey-homepage-vm/node_modules/@slack/webhook/src/errors.ts:59:5) [0-10] at IncomingWebhook.send (/builds/virginmediaO2/digital/hhh/journey-homepage-vm/node_modules/@slack/webhook/src/IncomingWebhook.ts:74:36) [0-10] at processTicksAndRejections (node:internal/process/task_queues:95:5)

[Data is Undefined] : DATA undefined

[0-11] 2023-07-25T09:56:37.939Z INFO @moroo/wdio-slack-reporter: POST Slack Request webhook:send [0-11] 2023-07-25T09:56:37.940Z DEBUG @moroo/wdio-slack-reporter: DATA undefined [0-11] 2023-07-25T09:56:38.080Z ERROR @moroo/wdio-slack-reporter: Error: An HTTP protocol error occurred: statusCode = 400 [0-11] at httpErrorWithOriginal (/builds/virginmediaO2/digital/hhh/journey-homepage-vm/node_modules/@slack/webhook/src/errors.ts:59:5) [0-11] at IncomingWebhook.send (/builds/virginmediaO2/digital/hhh/journey-homepage-vm/node_modules/@slack/webhook/src/IncomingWebhook.ts:74:36) [0-11] at processTicksAndRejections (node:internal/process/task_queues:95:5)

[sherryleneg]

@morooLee Thanks for getting back to me!

This is interesting - when I commented out this code snippet

//If spec file passes, do not post on Slack
        if (stateCounts.failed === 0) {
         return
        }

I was of course getting Slack notifications for both test passes and fails. However, the expected 'failing' test was being marked as passed, I'm not sure why that is. Isn't the test status determined by stateCounts.passed or stateCounts.failed?

This is a test notification (please ignore the undefined info, that's because some values are usually replaced by CI env vars).

What do you mean by: The problem seems to be happening on iOS when forking between iOS and Android.

[morooLee]

@sherryleneg Found the cause. I will fix it ASAP. I'll share it when it's done. Thank you!

[morooLee]

@sherryleneg Just updated. Please upgrade to 8.0.3. When using Cucumber, the state count was changed from scenario-based to test-based.

[sherryleneg]

@morooLee I can see the Slack notification for iOS now, however the passed and failed counts are still incorrect. It seems to be counting the steps rather than the scenario as a whole.

Should I use a different variable than stateCounts to get the scenario stats?

jobs-logs-ios-8.0.3.txt

[morooLee]

@sherryleneg Implemented by referring to the state count of Spec Reporter. Spec Reporter counts status on a based on th test(steps). However, we decided that it was necessary to count the state based on the scenario, so we added one useScenarioBasedStateCounts option. Now setting the value of useScenarioBasedStateCounts to true counts states based on the scenario.

• If even one of them fails, the scenario is counted as failed.
• If all are skipped, the scenario is counted as skipped.
• Default is counted as passed.

You can use it by upgrading to 8.1.0.

Configuration Example

export.config = {
  reporters: [
    [
      SlackReporter, {
        // Set the Slack Options used webhook.
        slackOptions: {
          type: 'webhook',
          webhook: process.env.SLACK_WEBHOOK_URL || "https://hooks.slack.com/........",
          slackName: "WebdriverIO Reporter",
          slackIconUrl: "https://webdriver.io/img/webdriverio.png",
        },
        // Set the Title of Test.
        title: 'Slack Reporter Test',
        // Set the Test Results URL.
        resultsUrl: process.env.JENKINS_URL,
        // Set the notification of Test Finished
        notifyTestFinishMessage: true,
        // Set the scenario-based state count (Only Cucumber)
        useScenarioBasedStateCounts: true,
        // Customize Slack Emoji Symbols.
        emojiSymbols: {
          passed: ':white_check_mark:',
          failed: ':x:',
          skipped: ':double_vertical_bar:',
          pending: ':grey_question:',
          start: ':rocket:',
          finished: ':checkered_flag:',
          watch: ':stopwatch:'
        },
        // Override the createStartPayload function.
        createStartPayload: function (runnerStats: RunnerStats): IncomingWebhookSendArguments {
          const payload: IncomingWebhookSendArguments = {
            // do something...
          }
          return payload;
        },
        // Override the createFailedTestPayload function.
        createFailedTestPayload: function (testStats: TestStats): IncomingWebhookSendArguments {
          const payload: IncomingWebhookSendArguments = {
            // do something...
          }
          return payload;
        },
        // Override the createResultPayload function.
        createResultPayload: function (runnerStats: RunnerStats, stateCounts: StateCount): IncomingWebhookSendArguments {
          const payload: IncomingWebhookSendArguments = {
            // do something...
          }
          return payload;
        }
      }
    ],
  ],
};

[sherryleneg]

@morooLee I tested with v8.1.0 and the Slack notifications works for both iOS and Android as expected. Thanks a lot!!