[QUESTION]:Builds/Workers not showing up properly in PW Portal

[imartinflores]

Hi guys, just a question not a bug, as I am not sure if this is correct behavior. I am running 2 jobs in parallel, each of them running X number of tests, using 10 workers. However, when I check the portal, seems like it is showing each test as an individual build. Instead of showing 2 different builds, is showing around 40, which is the amount of total tests.

This is what it's being run in the pipeline for 1 job:

$ npx playwright test --config=playwright.service.config.js --project=Android_Chrome_UK --workers=10 --grep @CN_UK

Running 20 tests using 10 workers
········°·°°·°··°···
  5 skipped
  15 passed (19.6s)
Done in 20.95s.

And if I check the portal, I can see each test as individual build, showing only 1 worker (So, 20 builds total). Is this a correct behavior ? It looks different to what I could see in another project.

Thanks!

AB#1909655

[puagarwa]

The possible explanation for this behavior is that your fixture is setting different runId for each worker/test , whereas ideally we should keep same runId during the run. please check it once.

If you see this line, we only set this env variable once and then onward dont update it during run. process.env.PLAYWRIGHT_SERVICE_RUN_ID = process.env.PLAYWRIGHT_SERVICE_RUN_ID || new Date().toISOString();

[imartinflores]

@puagarwa thanks. So that's what we were doing. The only thing we've changed is just adding the string at the beginning of the runId name. i.e: "Landers - new Date().toISOString();

The rest of the fixture is the same.. not sure if that can effect it. We reverted it back to the original state and will check.

/*
 * This file enables Playwright client to connect to remote browsers.
 * It should be placed in the same directory as playwright.config.ts.
 * The file is temporary for private preview.
 */

import { defineConfig } from '@playwright/test';
import config from './playwright.config';
import dotenv from 'dotenv';

// Define environment on the dev box in .env file:
//  .env:
//    PLAYWRIGHT_SERVICE_ACCESS_TOKEN=XXX
//    PLAYWRIGHT_SERVICE_URL=XXX

// Define environment in your GitHub workflow spec.
//  env:
//    PLAYWRIGHT_SERVICE_ACCESS_TOKEN: ${{ secrets.PLAYWRIGHT_SERVICE_ACCESS_TOKEN }}
//    PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
//    PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha }}

dotenv.config();

// Name the test run if it's not named yet.
process.env.PLAYWRIGHT_SERVICE_RUN_ID =
  process.env.PLAYWRIGHT_SERVICE_RUN_ID || `Landers - new Date().toISOString()`;

// Can be 'linux' or 'windows'.
const os = process.env.PLAYWRIGHT_SERVICE_OS || 'linux';

export default defineConfig(config, {
  // Define more generous timeout for the service operation if necessary.
  // timeout: 60000,
  // expect: {
  //   timeout: 10000,
  // },
  workers: 20,

  // Enable screenshot testing and configure directory with expectations.
  // https://learn.microsoft.com/azure/playwright-testing/how-to-configure-visual-comparisons
  ignoreSnapshots: false,
  snapshotPathTemplate: `{testDir}/__screenshots__/{testFilePath}/${os}/{arg}{ext}`,

  use: {
    // Specify the service endpoint.
    connectOptions: {
      wsEndpoint: `${process.env.PLAYWRIGHT_SERVICE_URL}?cap=${JSON.stringify({
        // Can be 'linux' or 'windows'.
        os,
        runId: process.env.PLAYWRIGHT_SERVICE_RUN_ID,
      })}`,
      timeout: 30000,
      headers: {
        'x-mpt-access-key': process.env.PLAYWRIGHT_SERVICE_ACCESS_TOKEN,
      },
      // Allow service to access the localhost.
      exposeNetwork: '<loopback>',
    },
  },
});

[imartinflores]

I can confirm is that, should there be any condition for this scenario ? seems like if any hardcoded name is being added to the runId, it takes it as a new runId every time, not sure if that's ideal..

thanks for looking into and the prompt response! Feel free to close it if need!

[puagarwa]

@imartinflores Any value should work here. You can put breakpoint to check why it is changing value everytime.

[puagarwa]

@imartinflores I used the above service config file and found one minor issue but i didnt face your issue, and able to see all test under same runId.

Minor issue is in this line, you need to add $() for Date, else it just resolve to constant.

process.env.PLAYWRIGHT_SERVICE_RUN_ID =
  process.env.PLAYWRIGHT_SERVICE_RUN_ID || `Landers - ${new Date().toISOString()}`;

Let me know if you still facing the issue. As i mentioned earlier you should be able to use any string for runId and all test for single run should come under single entry in activityLogs.

[imartinflores]

I'll give another try. It's weird, if you check my first screenshot, you will see they all say "TestRun#Landers - " and the correct date time.. but its generated 1 for each test. When I removed only the "Landers" from the name, it worked ok..

[puagarwa]

@imartinflores did you able to try it out, let me know if need any help here.

[imartinflores]

Hi @puagarwa sorry for the delay. I have not tried it again to be honest. The last time I included the string it was generating multiple builds so I only left the date() for now. Will try it again at some point and file a new issue if you prefer. Thanks for the support!