empty array reported when sending screenshots to slack

[rmkranack]

I am attempting to use the option to send the screen shots to Slack but The message in Slack is displaying an empty array. Can someone tell me if I am doing something incorrect?

//my_custom_layout.ts

import fs from "fs";
import path from "path";
import { Block, KnownBlock } from "@slack/types";
import { SummaryResults } from "playwright-slack-report/dist/src";
const web_api_1 = require('@slack/web-api');
const slackClient = new web_api_1.WebClient(process.env.SLACK_BOT_USER_OAUTH_TOKEN);

async function uploadFile(filePath) {
  try {
    const result = await slackClient.files.uploadV2({
      channels: 'automated-tests1',
      file: fs.createReadStream(filePath),
      filename: filePath.split('/').at(-1),
    });

   return result.file;
  } catch (error) {
    console.log('🔥🔥 error', error);
  }
}

export async function generateCustomLayoutAsync (summaryResults: SummaryResults): Promise<Array<KnownBlock | Block>> {
  const { tests } = summaryResults;

  const header = {
    type: "header",
    text: {
      type: "plain_text",
      text: "🎭 *Playwright E2E Test Results*",
      emoji: true,
    },
  };

  const summary = {
    type: "section",
    text: {
      type: "mrkdwn",
      text: `✅ *${summaryResults.passed}* | ❌ *${summaryResults.failed}* | ⏩ *${summaryResults.skipped}*`,
    },
  };

  const fails: Array<KnownBlock | Block> = [];

  for (const t of tests) {
    if (t.status === "failed" || t.status === "timedOut") {

     fails.push({
        type: "section",
        text: {
          type: "mrkdwn",
          text: `👎 *[${t.browser}] | ${t.suiteName.replace(/\W/gi, "-")}*`,
        },
      });

      const assets: Array<string> = [];

    if (t.attachments) {
      for (const a of t.attachments) {
        const file = await uploadFile(a.path);

        if (file) {
          if (a.name === 'screenshot' && file.permalink) {
            fails.push({
              alt_text: '',
              image_url: file.permalink,
              title: { type: 'plain_text', text: file.name || '' },
              type: 'image',
            });
          }

          if (a.name === 'video' && file.permalink) {
            fails.push({
              alt_text: '',
              // NOTE:
              // Slack requires thumbnail_url length to be more that 0
              // Either set screenshot url as the thumbnail or add a placeholder image url
              thumbnail_url: file.permalink,
              title: { type: 'plain_text', text: file.name || '' },
              type: 'video',
              video_url: file.permalink,
            });
          }
          if (assets.length > 0) {
            fails.push({
              type: "context",
              elements: [{ type: "mrkdwn", text: assets.join("\n") }],
            });
          }
        }
      }
    }

  }
}
  return [header, summary, { type: "divider" }, ...fails]
}

//playwright.config

...
   reporter: [
      [
        "./node_modules/playwright-slack-report/dist/src/SlackReporter.js",
        {
          channels: ["automated-tests1"], // provide one or more Slack channels
          sendResults: "always", // "always" , "on-failure", "off"
          layoutAsync: generateCustomLayoutAsync,
        },
      ],
   ],

  use: {
    screenshot: {
      mode: 'only-on-failure',
      fullPage: true,
    },
    video: "retain-on-failure",
...

[ryanrosello-og]

are you using projects within your playwright config?

this is where the browser is coming from https://github.com/ryanrosello-og/playwright-slack-report/blob/0557a7a8c8b0a9dc08d6d9e19b2ba9e2f0186827/src/ResultsParser.ts#L170

Bit tricky to troubleshoot, does this only affect failed tests or all tests?

could you try temporarily outputting everything? something like:

  for (const t of tests) {
    //if (t.status === "failed" || t.status === "timedOut") {

     fails.push({
        type: "section",
        text: {
          type: "mrkdwn",
          text: `👎 *[${t.browser}] | ${t.name}*`,
        },
      });
  }

[rmkranack]

are you using projects within your playwright config?

this is where the browser is coming from

https://github.com/ryanrosello-og/playwright-slack-report/blob/0557a7a8c8b0a9dc08d6d9e19b2ba9e2f0186827/src/ResultsParser.ts#L170

Bit tricky to troubleshoot, does this only affect failed tests or all tests?

could you try temporarily outputting everything? something like:

  for (const t of tests) {
    //if (t.status === "failed" || t.status === "timedOut") {

     fails.push({
        type: "section",
        text: {
          type: "mrkdwn",
          text: `👎 *[${t.browser}] | ${t.name}*`,
        },
      });
  }

I am using projects to configure the environments.

When I enable for passed tests, it does affect all tests.

[ryanrosello-og]

@rmkranack does your project config have the browser values specified? e.g.

    /* Test against branded browsers. */
    {
      name: 'Microsoft Edge',
      use: {
        channel: 'msedge',
      },
    },
    {
      name: 'Google Chrome',
      use: {
        channel: 'chrome',
      },
    },

[rmkranack]

I changed my config to have the browsers and still get the empty array. I also added the permission [files:write] to my Slack bot.

This is my current config file:

const { defineConfig, devices } = require('@playwright/test');
import { generateCustomLayoutAsync } from "./tests/utils/customLayout";

require('dotenv').config();

module.exports = defineConfig({
  testDir: './tests',
  /* Run tests in files in parallel */
  fullyParallel: true,
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.CI,
  /* Retry on CI only */
  retries: process.env.CI ? 2 : 0,
  /* Opt out of parallel tests on CI. */
  workers: process.env.CI ? 1 : undefined,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
  reporter: [
    [
      "./node_modules/playwright-slack-report/dist/src/SlackReporter.js",
      {
        channels: ["automated-tests1"], // provide one or more Slack channels
        sendResults: "always", // "always" , "on-failure", "off"
        layoutAsync: generateCustomLayoutAsync,
        slackOAuthToken: process.env.SLACK_BOT_USER_OAUTH_TOKEN,
      },
    ],
//   ['html'], // other reporters
 ],
  /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
  use: {
    /* Base URL to use in actions like `await page.goto('/')`. */
    //baseURL: process.env.QA_URL,
    screenshot: {
      mode: 'only-on-failure',
      fullPage: true,
    },
    video: "retain-on-failure",
    /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
    trace: 'on-first-retry',
  },
  globalSetup:"tests/utils/globalSetup.js",
  /* Configure projects for major browsers */
  projects: [
    {
      name: 'chrome',
      use: { ...devices['Desktop Chrome'] },
    },

   {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },

    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    },

    {
      name: 'chrome:latest:MacOS Monterey@lambdatest',
      use: {
        viewport: { width: 1920, height: 1080 }
      }
    },

    {
      name: 'chrome:latest:Windows 10@lambdatest',
      use: {
        viewport: { width: 1280, height: 720 }
      }
    }
  ],
});

[ryanrosello-og]

thanks for the extra info @rmkranack , I'll try and repro locally

[rmkranack]

I tried using the S3 setup and get the same result.

[ryanrosello-og]

hey @rmkranack Couple of things I noticed:

In your example, the config you specified in the playwright config file was complaining for me. I updated it from:

  use: {
    screenshot: {
      mode: 'only-on-failure',
      fullPage: true,
    },
    video: "retain-on-failure",

to

  use: {
    screenshot:'only-on-failure',
    video: "retain-on-failure",

After making the modifications above, I had to enable the files:write and files:read scope,

you will need to re-install the app and re-invite the bot into the channel (not sure why this was necessary to re-invite but it was the only way I could get it to work).

The next issue was the "channels", the example in the doco was using the channel name, but the api requires the channel id.

    const result = await slackClient.files.uploadV2({
      channels: 'automated-tests1',   <<< this needs to be the channel id
      file: fs.createReadStream(filePath),
      filename: filePath.split('/').at(-1),
    });

This channel id can be found in the url when you are in the channel. e.g.

https://app.slack.com/client/T02RVEEFPDH/C05H7TKVDUK

^ the bit starting with 'C...' is your channel id. In this case, the channel id is C05H7TKVDUK

After making the modifications above, I was able to send the message successfully ...

As a side note, I will need to make an update to the web-api version to suppress this warning:

[WARN] web-api:WebClient:0 Although the 'channels' parameter is still supported for smoother migration from legacy files.upload, we recommend using the new channel_id parameter with a single str value instead (e.g. 'C12345').

another issue - the attachments are being sent first and then the test summary message is being sent. I am not sure if this is the expected behaviour. I would expect the message the summary to be sent first and then the attachments. This is a separate piece of work to implement. I will raise a separate issue for this. fyi @IvanKalinin

[rmkranack]

Thanks for looking into this. I am seeing an error "Error in reporter Error: Channel ids [undefined] is not valid"

If it would be easier to get this working with S3, we can use that option. With the S3 option, this is what we are getting for the output

The custom layout is copied exactly

import fs from "fs";
import path from "path";
import { Block, KnownBlock } from "@slack/types";
import { SummaryResults } from "playwright-slack-report/dist/src";
import { PutObjectCommand, S3Client } from "@aws-sdk/client-s3";

const s3Client = new S3Client({
  credentials: {
    accessKeyId: process.env.S3_ACCESS_KEY || "",
    secretAccessKey: process.env.S3_SECRET || "",
  },
  region: process.env.S3_REGION,
});

async function uploadFile(filePath, fileName) {
  try {
    const ext = path.extname(filePath);
    const name = `${fileName}${ext}`;

    await s3Client.send(
      new PutObjectCommand({
        Bucket: process.env.S3_BUCKET,
        Key: name,
        Body: fs.createReadStream(filePath),
      })
    );

    return `https://${process.env.S3_BUCKET}.s3.${process.env.S3_REGION}.amazonaws.com/${name}`;
  } catch (err) {
    console.log("🔥🔥 Error", err);
  }
}

export async function generateCustomLayoutAsync (summaryResults: SummaryResults): Promise<Array<KnownBlock | Block>> {
  const { tests } = summaryResults;
  // create your custom slack blocks

  const header = {
    type: "header",
    text: {
      type: "plain_text",
      text: "🎭 *Playwright E2E Test Results*",
      emoji: true,
    },
  };

  const summary = {
    type: "section",
    text: {
      type: "mrkdwn",
      text: `✅ *${summaryResults.passed}* | ❌ *${summaryResults.failed}* | ⏩ *${summaryResults.skipped}*`,
    },
  };

  const fails: Array<KnownBlock | Block> = [];

  for (const t of tests) {
    if (t.status === "failed" || t.status === "timedOut") {

      fails.push({
        type: "section",
        text: {
          type: "mrkdwn",
          text: `👎 *[${t.browser}] | ${t.suiteName.replace(/\W/gi, "-")}*`,
        },
      });

      const assets: Array<string> = [];

      if (t.attachments) {
        for (const a of t.attachments) {
          // Upload failed tests screenshots and videos to the service of your choice
          // In my case I upload the to S3 bucket
          const permalink = await uploadFile(
            a.path,
            `${t.suiteName}--${t.name}`.replace(/\W/gi, "-").toLowerCase()
          );

          if (permalink) {
            let icon = "";
            if (a.name === "screenshot") {
              icon = "📸";
            } else if (a.name === "video") {
              icon = "🎥";
            }

            assets.push(`${icon}  See the <${permalink}|${a.name}>`);
          }
        }
      }

      if (assets.length > 0) {
        fails.push({
          type: "context",
          elements: [{ type: "mrkdwn", text: assets.join("\n") }],
        });
      }
    }
  }

  return [header, summary, { type: "divider" }, ...fails]
}

This is what is in the playwright.config.js

// @ts-check
const { defineConfig, devices } = require('@playwright/test');
import { generateCustomLayoutAsync } from "./src/tests/utils/customLayout";

/**
 * Read environment variables from file.
 * https://github.com/motdotla/dotenv
 */
require('dotenv').config();

/**
 * @see https://playwright.dev/docs/test-configuration
 */
module.exports = defineConfig({
  testDir: './src/tests',
  /* Run tests in files in parallel */
  fullyParallel: false,
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.CI,
  /* Retry on CI only */
  retries: process.env.CI ? 2 : 0,
  /* Opt out of parallel tests on CI. */
  workers: process.env.CI ? 1 : undefined,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
  reporter: [
    [
      "./node_modules/playwright-slack-report/dist/src/SlackReporter.js",
      {
        channels: ["automated-tests1"], // provide one or more Slack channels
        sendResults: "always", // "always" , "on-failure", "off"
        layoutAsync: generateCustomLayoutAsync,
        slackOAuthToken: process.env.SLACK_BOT_USER_OAUTH_TOKEN,
      },
    ],
  ['dot'], 
],
timeout: 60000,
  use: {
    screenshot: 'only-on-failure',
    video: "retain-on-failure",
    trace: 'on-first-retry',
  },
  /* Configure projects for major browsers */
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },

    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },

    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    },

    {
      name: 'chrome:latest:MacOS Monterey@lambdatest',
      use: {
        viewport: { width: 1920, height: 1080 }
      }
    },

    {
      name: 'chrome:latest:Windows 10@lambdatest',
      use: {
        viewport: { width: 1280, height: 720 }
      }
    },
  ],
});

[ryanrosello-og]

hey @rmkranack , are you running this using v1.1.20 (the latest version)

[rmkranack]

I am. Here are the versions of what I am running.

[ryanrosello-og]

Made progress:

but it needs a bit more testing to make sure I haven't broken it for people that do not use "projects" in their playwright config and for those people that spawn their own browsers

[ryanrosello-og]

@rmkranack can you give 1.1.21 a try

[rmkranack]

The screenshots and video upload from S3 is working! Great job! Is this supposed to also output the failure reason or does that not work with the screenshot upload option?

[ryanrosello-og]

Is this supposed to also output the failure reason or does that not work with the screenshot upload option?

You'll need to update your generateCustomLayoutAsync to enable this.

You have this entire object to play around with

export type SummaryResults = {
  passed: number;
  failed: number;
  skipped: number;
  failures: Array<failure>;
  meta?: Array<{ key: string; value: string }>;
  tests: Array<{
    suiteName: string;
    name: string;
    browser?: string;
    projectName?: string;
    endedAt: string;
    reason: string;
    retry: number;
    startedAt: string;
    status: 'passed' | 'failed' | 'timedOut' | 'skipped';
    attachments?: {
      body: string | undefined | Buffer;
      contentType: string;
      name: string;
      path: string;
    }[];
  }>;
};

Specifically, look at the failures array which contains the failure reason

See example here ... https://github.com/ryanrosello-og/playwright-slack-report/blob/main/src/custom_block/my_block.ts#L12

[rmkranack]

I will take a look. Thanks again for your help!