[BUG]: PLAYWRIGHT_SERVICE_RUN_ID no longer supported?

[cthlo]

Describe the bug Since 1.0.0-beta.4, the test run ID can no longer be set with PLAYWRIGHT_SERVICE_RUN_ID.

To Reproduce Steps to reproduce the behavior:

1. Set env var PLAYWRIGHT_SERVICE_RUN_ID=some-test-run-id
2. Run tests
3. Resulting test run ID is not some-test-run-id

Expected behavior PLAYWRIGHT_SERVICE_RUN_ID controls run ID. If not, please update documentations:

• https://learn.microsoft.com/en-us/azure/playwright-testing/quickstart-automate-end-to-end-testing
• https://learn.microsoft.com/en-us/azure/playwright-testing/playwright-testing-reporting-with-sharding

Screenshots If applicable, add screenshots to help explain your problem.

Setup information (please complete the following information):

• Azure Subscription Id:
• Workspace Name:
• Service trace Id (if available):
• Log snippet (if available):
• Browser on which tests are executed (if applicable):
• Time range (in PST) when the issue was experienced:
• Playwright Version:
  • Client machine OS:
  • OS requested on Service:

Additional context

Due to this? https://github.com/Azure/azure-sdk-for-js/commit/497321f0400ba4a2adfbd8f59db62d7574b52824

Thanks!

[vvs11]

Hi @cthlo Thanks for writing to us. We don't have the variable PLAYWRIGHT_SERVICE_RUN_ID in the service package. You can still use it by adding run id in the playwright.service.config.ts file like this:

runId:process.env.PLAYWRIGHT_SERVICE_RUN_ID

You can then define PLAYWRIGHT_SERVICE_RUN_ID as environment variable. I will update the docs to add this line in playwright.service.config.ts file

[cthlo]

Hi @cthlo Thanks for writing to us. We don't have the variable PLAYWRIGHT_SERVICE_RUN_ID in the service package. You can still use it by adding run id in the playwright.service.config.ts file like this:

runId:process.env.PLAYWRIGHT_SERVICE_RUN_ID

You can then define PLAYWRIGHT_SERVICE_RUN_ID as environment variable. I will update the docs to add this line in playwright.service.config.ts file

Thanks for the pointer, @vvs11. I didn't realize I could use runId. I was just a little confused as PLAYWRIGHT_SERVICE_RUN_ID used to work prior to the 1.0.0-beta.4. I'm totally okay with using runId now, but was just curious if it was intentionally removed.

Thanks!

Per the documentations,

Quickstart: Set up continuous end-to-end testing with Microsoft Playwright Testing Preview

playwright.service.config.ts:

import { defineConfig } from '@playwright/test';
import { getServiceConfig, ServiceOS } from '@azure/microsoft-playwright-testing';
import config from './playwright.config';

/* Learn more about service configuration at https://aka.ms/mpt/config */
export default defineConfig(
  config,
  getServiceConfig(config, {
    exposeNetwork: '<loopback>',
        timeout: 30000,
    os: ServiceOS.LINUX,
        useCloudHostedBrowsers: true
  }),
  {
    /* 
    Playwright Testing service reporter is added by default.
    This will override any reporter options specified in the base playwright config.
    If you are using more reporters, please update your configuration accordingly.
    */
    reporter: [['list'], ['@azure/microsoft-playwright-testing/reporter']],
  }
);

GitHub Actions:

- name: Run Playwright tests
    working-directory: path/to/playwright/folder # update accordingly
  env:
    # Regional endpoint for Microsoft Playwright Testing
    PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
    PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha }}
  run: npx playwright test -c playwright.service.config.ts --workers=20

Use Microsoft Playwright Testing Reporting with Playwright sharding (preview)

You can use Playwright Testing's reporting feature to get a consolidated report of a test run with sharding. You need to make sure you set the variable PLAYWRIGHT_SERVICE_RUN_ID so that it remains same across all shards.

The PLAYWRIGHT_SERVICE_RUN_ID variable is an identifier that is used by Playwright Testing service to distinguish between test runs. The results from multiple runs with same RUN_ID are reported to the same run on the Playwright portal.

By default, a test run that uses reporting feature automatically generates a unique RUN_ID unless you explicitly set the value yourself. If the value of the variable remains same across runs, the results are reported together in the same run on the Playwright portal.