[BUG] Can't record trace in a test that opens a download/pdf file

[mastacheata]

System info

• Playwright Version: v1.36.0
• Operating System: Windows 11 (WSL Ubuntu)
• Browser: Chromium
• Other info:

Config file

import { defineConfig } from '@playwright/test'
import { devices } from '@playwright/test'

/**
 * See https://playwright.dev/docs/test-configuration.
 */
export default defineConfig({
  testDir: './playwright-e2e',

  /**
   Maximum time one test can run for => 120 seconds.
   This timeout is also the reason, why the playwright generator exits after 120 seconds (120 * 1000).
   You need to temporarily adjust this value if you want to use the generator for a longer time.
   */
  // timeout: 120 * 1000 * 1000,
  timeout: 120 * 1000,

  expect: {
    /**
     * Maximum time expect() should wait for the condition to be met.
     * For example in `await expect(locator).toHaveText();`
     */
    timeout: 100000,
  },

  /* Report the slowest tests (up to `max`) slower than `threshold` (in ms) */
  reportSlowTests: { max: 5, threshold: 30 * 1000 },

  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.CI,

  /* Retry on CI only */
  retries: process.env.CI ? 2 : 0,

  /*
   * Use only as many workers as there are CPUs, but at most 4.
   *
   * Ambient Gitlab-CI workers in May 2022 crashed consistently with more than 2 worker nodes.
   * On an AMD machine with 6 cores and 12 threads using more than 6 workers slowed the tests significantly
   * Because 4-cores/8-threads is the most common dev machine, using 4 workers at most seems reasonable
   */
  workers: process.env.CI ? 2 : 4,

  /*
   * Run tests inside a Testcase in parallel by default.
   * Use test.describe.configure({ mode: 'serial' }) to avoid that,
   * but at this point in time all our tests were  manually marked as parallel mode anyway.
   */
  fullyParallel: true,

  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
  reporter: process.env.CI
    ? [['junit', { outputFile: 'test-results/junit.xml' }], ['list']]
    : 'list',

  /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
  use: {
    /* Maximum time each action such as `click()` can take. Defaults to 0 (no limit). Limit set to 30s to stop
     infinite tests */
    actionTimeout: 30 * 1000,

    /* Base URL to use in actions like `await page.goto('/')`. */
    baseURL: 'http://localhost:3000',

    screenshot: process.env.PLAYWRIGHT_SLOW_MO ? 'on' : 'only-on-failure',
    video: process.env.PLAYWRIGHT_SLOW_MO ? 'on' : 'retry-with-video',
    trace: process.env.PLAYWRIGHT_SLOW_MO ? 'on' : 'on-all-retries',
  },

  /* Configure projects for major browsers */
  projects: [
    {
      name: 'empto',

      /*
      Project-specific settings.

      launchOptions: {slowMo: 1000,} controls how fast playwright executes the specified commands in a headed run.
      Uncomment this option, if you want to have time when watching how playwright executes tests
      */
      use: {
        ...devices['Desktop Chrome'],
        viewport: { width: 1440, height: 900 },
        launchOptions: {
          slowMo: parseInt(process.env.PLAYWRIGHT_SLOW_MO || '0'),
        },
      },
    },
  ],

  /* Folder for test artifacts such as screenshots, videos, traces, etc. */
  outputDir: 'test-results/',
  globalSetup: require.resolve('./global-setup'),
})

Test file (self-contained)

test.describe('Try valid users', () => {
    test.use({
      storageState: `loginContexts/loggedin.json`,
    })
    test(`are the internal downloads visible for loggedin`, async ({
      page,
      headless,
    }) => {
      await page.goto('http://localhost:3000/')
      await page.locator('.dropdown-toggle-icon >> nth=-1').click()
      await page.getByRole('link', { name: 'Interne Downloads' }).click()
      await page.getByRole('link', { name: 'Testseite' })

      if (headless) {
        // PDF Files are downloaded in headless mode
        const downloadPromise = page.waitForEvent('download')
        await page
          .locator('a:text("Testseite"):above(a:text("epd Logo vertical"))')
          .click()
        const download = await downloadPromise
        const filename = download.suggestedFilename()
        await expect(filename).toBe('testseite.pdf')
      } else {
        // PDF files are shown in a new browser tab using the built-in PDF viewer in headed mode
        const popupPromise = page.waitForEvent('popup')
        await page
          .locator('a:text("Testseite"):above(a:text("epd Logo vertical"))')
          .click()
        const popup = await popupPromise
        // Wait for the popup to load.
        await popup.waitForLoadState()
        const url = popup.url()
        await expect(url).toContain('testseite.pdf')
      }
    })
  })

Steps

• Run the test with traces set to always record (on)

Expected

• The test should work just fine even if I tell it to record traces on successful runs

Actual

• The test fails, complaining it can't find screenshot #2 Error: ENOENT: no such file or directory, open '/home/benedikt/develop/empto/test-results/general-emptoInternalDownloads-Try-valid-users-6acb2-ads-visible-for-mp-makler-premium-dashboard-epd-empto/test-finished-2.png'

• If I turn off the trace-recording and only save a video and screenshots the test runs fine

• If I run the test in headed mode trace-recording works just fine as well. This leads me to believe the problem is with taking a screenshot of a Download (headless mode) vs taking a screenshot of the PDF-Viewer Tab that chrome opens in headed mode instead.

I can't share the site the test is running against (proprietary code, not hosted on GitHub) - If absolutely necessary to reproduce, I can try and whip up a minimal sample site.
The testcase explicitly has separate paths for headless and headed mode because Playwright has no way to turn on/off browser features (here: the PDF-Viewer) in either headless or headed mode. (AFAIK there is no way to enable the PDF viewer in chromium headless)
I'm aware this isn't a great solution, but it's one that works (well, except for the trace-recorder, which seems to have problems here)

[pavelfeldman]

I would guess that the problem is with the screenshot: option - the trace entry for it is generated, but the screenshot is not actually taken because the page is not a page, but a pdf viewer plugin. Could you confirm that removing screenshot: helps?

[mastacheata]

Ohh, interesting.

Yes, screenshot and trace are mutually exclusive for the headless test.
Disabling either yields the exepected results for the other.

Only when enabling both at once it crashes.