[BUG] page goto failed: page was closed error running parallel sequential tests in Chromium against local html file

[GrayedFox]

System info

• Playwright Version: [v1.33.0]
• Operating System: Ubuntu 22.04.2 LTS
• Browser: Chromium
• Other info: Confirmed on Chromium but I think it likely affects all browsers

Note: OP has been edited for clarity.

Source code

implementation details

// open browser, inject needed Carted window object, and wait for graphQL responses
async open(waitForApi = true) {
  const graphQlResponses = waitForResponses(this.page, 4);

  await this.injectCarted();
  // note no await
  this.page.goto(ClientData.baseUrl + '/index.html');

  // workarounds that fix the issue:
  // 1. await this.page.goto(ClientData.baseUrl + '/index.html');
  // 2. this.page.goto(ClientData.baseUrl + '/index.html', { waitUntil: 'commit' });

  if (waitForApi) {
    await graphQlResponses;
  }
}

// example page object class getter - `this.self` is itself a locator that scopes
// the CSS query, while OrderStatusCardSelectors.confirmationHeader is a CSS string
get confirmationHeader() {
  return this.self.locator(OrderStatusCardSelectors.confirmationHeader);
}

test.spec.ts

import { Page, test } from '@playwright/test';
import { Browser, Page } from '@playwright/test';

// graphql helper
import CartedSession from '../../graphql';
// extended test and expect
import { expect, test } from '../fixtures';
// page objects
import {
  CreditCard,
  FloatingCartButton,
  OrderStatusCard,
  ShoppingCart,
} from '../page-objects';
// support
import { cleanUp, getState, sendResults, setUp } from '../support';
// test data
import {
  adidasBackpack,
  ccDetailsTestCard,
  plushToy,
  staticUSShippingDetails,
} from '../test-data';

test.describe.configure({ mode: 'serial' });

test.describe('Order Confirmation', () => {
  const cartedSession = new CartedSession('order-confirmation');
  let page: Page;
  let testBrowser: Browser;
  let cart: ShoppingCart;
  let floatingCartButton: FloatingCartButton;
  let payment: CreditCard;
  let orderStatus: OrderStatusCard;

  test.beforeAll(async ({ browser }, testInfo) => {
    // use the GraphAPI directly set session state, loading needed fixtures (if any)
    // all these requests happens before the browser even opens, so we can seed some state
    await cartedSession.loadFixture();
    await cartedSession.fetchProduct(plushToy);
    await cartedSession.fetchProduct(adidasBackpack);
    await cartedSession.addToCart(plushToy);
    await cartedSession.addToCart(adidasBackpack, 2);
    await cartedSession.startCheckout();
    await cartedSession.setAddress(staticUSShippingDetails);
    await cartedSession.setShippingOptions([
      { vendorName: plushToy.vendor, label: 'Economy' },
      { vendorName: adidasBackpack.vendor, label: 'Standard' },
    ]);

    // fetch local storage state object
    const state = getState([cartedSession.localStorageObject]);
    const mockOptions = {
      fixtureName: 'order-confirmation',
      storageState: state,
      sessionId: cartedSession.sessionId,
    };

    // now we create a page and setup calls
    // browser.newPage({ storageState }) at some point
    [page, testBrowser] = await setUp(browser, testInfo, mockOptions);

    // init page objects
    cart = new ShoppingCart(page);
    floatingCartButton = new FloatingCartButton(page);
    payment = new CreditCard(page);
    orderStatus = new OrderStatusCard(page);

    // launch the browser, open the cart, confirm payment
    await payment.open();
    await floatingCartButton.openCart();
    await payment.enterCreditCardDetails(ccDetailsTestCard);
    await cart.confirmPurchase();
  });

  test.afterEach(async ({}, testInfo) => {
    await sendResults(page, testInfo);
  });

  test.afterAll(async ({}) => {
    await cleanUp(page, testBrowser, { fixtureName: 'order-confirmation' });
  });

  test('shows the order confrimation header', () => {
    expect(orderStatus.confirmationHeader).toBeVisible();
  });

  test('displays correct email in thank you summary', () => {
    expect(orderStatus.emailSummary).toContainText(
      staticUSShippingDetails.email
    );
  });

  test('displays Order Placed for the first item', () => {
    // set order status card to equal plushtoy vendor
    orderStatus.merchantName = plushToy.vendor;
    expect(orderStatus.badge).toHaveText('Order Placed');
  });

  test('displays correct title for first item', () => {
    expect(orderStatus.title).toHaveText(plushToy.title);
  });

  test.fixme('displays correct image for first item ', () => {
    expect(orderStatus.image).toHaveScreenshot();
  });

  test('displays correct vendor name for first item ', () => {
    expect(orderStatus.vendor).toContainText(plushToy.vendor);
  });

  test('displays Order Placed for the second item', () => {
    // set order status card to equal backpack vendor
    orderStatus.merchantName = adidasBackpack.vendor;
    expect(orderStatus.badge).toHaveText('Order Placed');
  });

  test('displays correct title for second item', () => {
    expect(orderStatus.title).toHaveText(adidasBackpack.title);
  });

  test('displays correct vendor name for second item ', () => {
    expect(orderStatus.vendor).toContainText(adidasBackpack.vendor);
  });

  test.fixme('displays correct image for second item ', () => {
    expect(orderStatus.image).toHaveScreenshot();
  });
});

index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <meta name="description" content="Test File" />
    <title>Publisher Page</title>
    <script>
      // if thing defined don't overwrite it
      if (!window['thing']) {
          window['thing'] = {
          environment: 'sandbox',
          apiKey: 'default',
          onMessage: console.debug,
          displayName: 'PlayWright Tester',
        };
      }
      // actual index test file imports a module using scripts src='url' defer
      // this module registers a custom component which calls our graphQL api 
  </script>
    <style>
      * { box-sizing: border-box; }
      body { margin: 0; padding: 2rem; }
      main { max-width: min(100%, 58rem); margin: 0 auto; }
      carted-product-card { contain: size layout paint; }
    </style>
  </head>
  <body>
    <main>
      <article>

        <h2>Hello world!</h2>
        <p>This is me, life can be: fun for everyone</p>
        <custom-component src='url'></custom-component>
      </article>
    </main>
  </body>
</html>

• [x] I provided exact source code that allows reproducing the issue locally.

Config file

// playwright.config.ts
import type { PlaywrightTestConfig } from '@playwright/test';
import dotenv from 'dotenv';

// read .env file values
dotenv.config();

// this config sets the browserstack env var to false
process.env.BROWSERSTACK = 'false';

const config: PlaywrightTestConfig = {
  name: 'Local Config',
  globalSetup: './playwright/global-setup.ts', // only thing this does is remove all files in the dist directory
  testDir: './playwright/integration',
  timeout: 100_000,
  reporter: [['html', { outputFolder: 'dist/playwright/reports' }]],
  outputDir: 'dist/playwright/test-results/',
  retries: 0,
  expect: {
    timeout: 10_000,
    toHaveScreenshot: {
      threshold: 0.2,
      maxDiffPixelRatio: 0.1,
      maxDiffPixels: 5,
    },
  },
  use: {
    actionTimeout: 0,
    screenshot: 'only-on-failure',
    trace: 'on',
    video: 'off',
    contextOptions: {
      recordVideo: {
        dir: 'dist/playwright/videos',
      },
    },
  },
  projects: [
    {
      name: 'chromium',
      use: {
        browserName: 'chromium',
        channel: 'chromium',
      },
    },
  ],
};

export default config;
});

Steps

1. Run the test
2. I rarely see this bug occur when running a single test, create a bunch of dummy test files and run them in parallel to boost the chances of seeing the bug

Expected

No "page goto failed: page was closed error" is thrown

Actual

Fails 33% of the time on my machine. I don't know for sure but after verifying the valid load state here it seems like eventually the private server _goToAction method will be called, which on line 693 awaits an extra LifecycleEvent event promise.

My suspicion is that the file protocol is behaving differently (due to how the browser treats local html files) - or that it is perhaps just faster than the typical http/https file that, under normal circumstances, is served up by a server which is therefore exposing some raciness.

Mileage may vary, I believe the above code reproduces the issue, I had to take a whole lot of IP and other work related stuff out that over complicates things, but wanted to leave stuff like dotenv to mirror our code as close as possible.

What I can confirm is that setting the wait until option to commit fixes the issue for us.

[yury-s]

If there is no await before page.goto you may well get the exception because the context and the page are closed after the test, before load event is fired. You maybe more lucky with waitUntil: 'commit' just because 'commit' event is fired much sooner. If you omit await before page.goto this is working as intended, you should await for the action to avoid the error.

[GrayedFox]

Hi there @yury-s - thanks for looking into this. Rather than await the goto I am using a custom function to await the network requests to our GraphQL backend made by the page that is navigated to, since that it what signals a "ready to be tested" state. If I await the goto, the next promise (which awaits the requests) can be too slow and miss a request. I know I can do Promise.all() and do so when combining a Locator method (like a button click) with triggers a request, but technically don't need that here.

Below is the actual method we use at the beginning of each test - the injectCarted() method exposes a function as well as injects an init script -- very similar to what is in the above OP -- and then we navigate using goto and immediately after await some GraphQL responses.

// open browser, inject needed Carted window object, and wait for graphQL responses
async open(waitForApi = true) {
  await this.injectCarted();
  this.page.goto(ClientData.baseUrl + '/index.html');
  if (!waitForApi) return;
  await waitForResponses(this.page, 4);
}

All of the intermittently failing tests in question wait for 4-6 graphql requests to finish - if I edit the code sample in the OP to await for 2000ms before returning a promise after navigating to the page, and the same bug occurs, would you consider that proof of an issue?

[yury-s]

and then we navigate using goto and immediately after await some GraphQL responses.

This is racy. You start waiting when some of the requests may have been sent. You should start waiting before the navigation:

// open browser, inject needed Carted window object, and wait for graphQL responses
async open(waitForApi = true) {
  await this.injectCarted();
  const receivedApiResponses = waitForApi ? waitForResponses(this.page, 4) : undefined;
  await this.page.goto(ClientData.baseUrl + '/index.html');
  await receivedApiResponses;
}

This way you ensure that there is no unhandled promise from page.goto and that the page didn't issue any requests before you started listening for them. You can pass waitUntil: 'commit' if you like since you have an alternative condition for the page readiness. If you don't await for page.goto, you have to catch potential error in page.goto as you don't know if the page will be closed before the navigation completes.

[GrayedFox]

Alright so, if I refactor the code to capture the responses first but don't await the goto:

// open browser, inject needed Carted window object, and wait for graphQL responses
async open(waitForApi = true) {
  const graphQlResponses = waitForResponses(this.page, 4);

  await this.injectCarted();
  this.page.goto(ClientData.baseUrl + '/index.html');

  if (waitForApi) {
    await graphQlResponses;
  }
}

Would you consider that proof of a bug? Note I will need to test it still occurs and post another response here. Again thanks for your time/

Edit: ternary version you posted is better as it doesn't unnecessarily create a promise that goes unused - just a style choice on my end 👍🏽

[yury-s]

Would you consider that proof of a bug?

You still have to either await this.page.goto or this.page.goto(ClientData.baseUrl + '/index.html').catch(() => {}); if the page can close before the navigation finishes. An uncaught exception from pending page.goto thrown during context closure is not a bug.

[GrayedFox]

Okay I think I perhaps haven't explained the issue very well.

The problem is, it seems like the page/context closure is happening too soon - in particular - it is happening before tests have finished running.

The page shouldn't close before navigation finishes, because the page and context should remain open until the cleanUp call during the afterAll hook.

It looks like the goto method is throwing before any tests have been run (afaik). The bug, then, is not that goto throws due to the context or page being closed - but rather - that the page is being closed before the test run finishes (or even really begins).

I can confirm that if I don't await the goto method, even when capturing responses before making the page navigation, I get the navigation failed due to page being closed error - and even when there are other Locator methods after the initial page navigation that are awaiting promises.

Here is a full test file which exhibits the bug:

Edit: this is just the same code posted in the OP now

Order confirmation spec

```ts import { Browser, Page } from '@playwright/test'; // graphql helper import CartedSession from '../../graphql'; // extended test and expect import { expect, test } from '../fixtures'; // page objects import { CreditCard, FloatingCartButton, OrderStatusCard, ShoppingCart, } from '../page-objects'; // support import { cleanUp, getState, sendResults, setUp } from '../support'; // test data import { adidasBackpack, ccDetailsTestCard, plushToy, staticUSShippingDetails, } from '../test-data'; test.describe.configure({ mode: 'serial' }); test.describe('Order Confirmation', () => { const cartedSession = new CartedSession('order-confirmation'); let page: Page; let testBrowser: Browser; let cart: ShoppingCart; let floatingCartButton: FloatingCartButton; let payment: CreditCard; let orderStatus: OrderStatusCard; test.beforeAll(async ({ browser }, testInfo) => { // use the GraphAPI directly set session state, loading needed fixtures (if any) // all these requests happens before the browser even opens, so we can seed some state await cartedSession.loadFixture(); await cartedSession.fetchProduct(plushToy); await cartedSession.fetchProduct(adidasBackpack); await cartedSession.addToCart(plushToy); await cartedSession.addToCart(adidasBackpack, 2); await cartedSession.startCheckout(); await cartedSession.setAddress(staticUSShippingDetails); await cartedSession.setShippingOptions([ { vendorName: plushToy.vendor, label: 'Economy' }, { vendorName: adidasBackpack.vendor, label: 'Standard' }, ]); // fetch local storage state object const state = getState([cartedSession.localStorageObject]); const mockOptions = { fixtureName: 'order-confirmation', storageState: state, sessionId: cartedSession.sessionId, }; // now we create a page and setup calls // browser.newPage({ storageState }) at some point [page, testBrowser] = await setUp(browser, testInfo, mockOptions); // init page objects cart = new ShoppingCart(page); floatingCartButton = new FloatingCartButton(page); payment = new CreditCard(page); orderStatus = new OrderStatusCard(page); // launch the browser, open the cart, confirm payment await payment.open(); await floatingCartButton.openCart(); await payment.enterCreditCardDetails(ccDetailsTestCard); await cart.confirmPurchase(); }); test.afterEach(async ({}, testInfo) => { await sendResults(page, testInfo); }); test.afterAll(async ({}) => { await cleanUp(page, testBrowser, { fixtureName: 'order-confirmation' }); }); test('shows the order confrimation header', () => { expect(orderStatus.confirmationHeader).toBeVisible(); }); test('displays correct email in thank you summary', () => { expect(orderStatus.emailSummary).toContainText( staticUSShippingDetails.email ); }); test('displays Order Placed for the first item', () => { // set order status card to equal plushtoy vendor orderStatus.merchantName = plushToy.vendor; expect(orderStatus.badge).toHaveText('Order Placed'); }); test('displays correct title for first item', () => { expect(orderStatus.title).toHaveText(plushToy.title); }); test.fixme('displays correct image for first item ', () => { expect(orderStatus.image).toHaveScreenshot(); }); test('displays correct vendor name for first item ', () => { expect(orderStatus.vendor).toContainText(plushToy.vendor); }); test('displays Order Placed for the second item', () => { // set order status card to equal backpack vendor orderStatus.merchantName = adidasBackpack.vendor; expect(orderStatus.badge).toHaveText('Order Placed'); }); test('displays correct title for second item', () => { expect(orderStatus.title).toHaveText(adidasBackpack.title); }); test('displays correct vendor name for second item ', () => { expect(orderStatus.vendor).toContainText(adidasBackpack.vendor); }); test.fixme('displays correct image for second item ', () => { expect(orderStatus.image).toHaveScreenshot(); }); }); ```

I get that a bunch of the implementation details are hidden - but I think this still hopefully clarifies what I think the issue is. Note that even when payment.open() calls the exact code I posted just before (with the graphQlResponses declared before the goto) -- and proceeds to interact with the page by filling out a form and submitting it - the error will occur.

What I don't get is what, exactly, is causing the page to close before the goto method has had a chance to navigate when we are awaiting further promises after the goto? The cleanup method shouldn't be called until after all tests run and those tests shouldn't be run until the entire beforeAll hook has finished.

It seems like page goto, when not awaited, is buggy in that it somehow causes a page or context to be closed when it shouldn't be (or perhaps goes to navigate before the page is fully opened, thus returning a cryptic error making it look like the page/context has been closed, but really the page hasn't fully opened yet -- assuming closed is the same as the initial state).

I will update the OP so the code is clearer.

[yury-s]

It looks like the goto method is throwing before any tests have been run (afaik). The bug, then, is not that goto throws due to the context or page being closed - but rather - that the page is being closed before the test run finishes (or even really begins).

All the tests are missing await before toHaveText, toHaveScreenshot, toContainText calls. Locator assertions are asynchronous if you don't await them then the test runner may close the page before the actual assertion finishes the check. Make sure that you await all asynchronous calls in the playwright API, otherwise you may easily get errors like you mentioned. You may want to run an audit to check that there are no hanging async calls.

[yury-s]

Closing per the response above, feel free to open a new issue if it doesn't work.

[GrayedFox]

Confirming that removing the waitUntil option and instead awaiting each expect from inside the tests fixed it.

I recently converted the whole test suite from Cypress to PW and that was a relic I totally overlooked 🤦🏽

Thanks for helping me get to the bottom of the error, hopefully future eyeballs will benefit if they come across this!