[Question] Storage State Re-Use

[miguelofbc]

Context:

• Playwright Version: 1.23.1
• Operating System: macOS Monterey 12.4 (Apple M1)
• Node.js version: 16.15.0
• Browser: All
• Extra: NA

Code Snippet

// playwright.config.ts
import { PlaywrightTestConfig, devices } from "@playwright/test";
(...)

const config: PlaywrightTestConfig = {
(...)
 globalSetup: require.resolve("./playwright/global-setup"),
 use: {
(...)
    storageState: "playwright/fixtures/storageState.json",
  },
  projects: [
   (...)
  ],
};
export default config;

// global-setup.ts
import { chromium, FullConfig } from "@playwright/test";
import path from "path";

async function globalSetup(config: FullConfig) {
  console.log("Global Setup is running...");
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();
  await context.tracing.start({ screenshots: true, snapshots: true });
  await page.goto(`${process.env.BASEURL}`);
  await page.fill('input[name="username"]', `${process.env.PW_USERNAME}`);
  await page.fill('input[name="password"]', `${process.env.PW_PASSWORD}`);
  await page.click("text=Sign in");
  // // Save signed-in state to 'storageState.json'.
  const __dirname = path.resolve("playwright/fixtures/storageState.json");
  console.log("Current directory:", __dirname);
  await page
    .context()
    .storageState({ path: "playwright/fixtures/storageState.json" });
  // await browser.close();
  await page.close();
  await context.tracing.stop({ path: "trace.zip" });
  console.log("Global Setup ended!");
}

export default globalSetup;

// baseFixtures.ts
import { test as base, chromium } from "@playwright/test";
import { LoginPage } from "../pom/login-page";

// Declare the types of your fixtures.
type CommonFixtures = {
  loginPage: LoginPage;
  homePage: HomePage;
};

export const test = base.extend<CommonFixtures>({
  loginPage: async ({ page }, use) => {
    const loginPage = new LoginPage(page);
    await use(loginPage);
  },
  homePage: async ({ page }, use) => {
    const homePage = new HomePage(page);
    await use(homePage);
  },
});

export default test;
export { expect } from "@playwright/test";

// auth.spec.ts
import { test } from "../../framework/baseFixtures";
import path from "path";

test.beforeEach(async ({ page }, testInfo) => {
  console.log(
    `Running beforeEach on worker ${testInfo.workerIndex} for ${testInfo.title}`
  );
  await page.goto("/");
});

test.afterAll(async ({ page }, testInfo) => {
  console.log(
    `Running afterAll on worker ${testInfo.workerIndex} for test "${testInfo.title}" with status=${testInfo.status}`
  );

  if (testInfo.status !== testInfo.expectedStatus)
    console.log(`Did not run as expected, ended up at ${page.url()}`);
});

test.describe("Authentication", () => {
  // test("User can login", async ({ loginPage, homePage, page }) => {
  //   await loginPage.login(process.env.PW_USERNAME, process.env.PW_PASSWORD);
  //   await homePage.userIsHomePage();
  // });
  test("User can Logout", async ({ homePage, logoutPage }) => {
    await homePage.userIsHomePage();
    await homePage.logout();
    await logoutPage.userIsLoggedOut();
  });
});

test.describe("Negative test", () => {
  // make sure user is not authenticated by forcing empty storageState
  test.use({ storageState: "playwright/fixtures/userStorageState.json" });
  test("User can't login with invalid credentials", async ({
    loginPage,
    logoutPage,
  }) => {
    await loginPage.login("wrong", "password");
    await loginPage.wrongCredentialsValidation();
    await logoutPage.userIsLoggedOut();
  });
});

Describe the bug

I am facing some issues when trying to implement reuse signed in state with fixtures to test an application with Keycloak (KC) authentication. By comparing a successful authentication using the storageState with Trace, I've been able to understand that the cookie token was not valid when the test was running. After that, I realise that the KC auth session was not active after the global-setup execution. As a workaround, it was needed to login inside the beforeEach or beforeAll hook or inside the test.describe, save the storageState to the context and only after that for the remaining tests in the test.describe, the authentication was working. Caveat: the sessions aren't being closed after each test run.

test.beforeAll(async ({}, testInfo) => {
  console.log(`Running beforeAll on worker ${testInfo.workerIndex}`);
  console.log(
    `Creating Keycloak session and storing cookies under ${__storageStateDir}`
  );

  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();
  await page.goto("/");
  loginPage = new LoginPage(page);
  await loginPage.login(process.env.PW_USERNAME, process.env.PW_PASSWORD);
  await page
    .context()
    .storageState({ path: "playwright/fixtures/storageState.json" });

  await page.close();
  await context.close();
});

1. Is there any example that I can use as a reference on how to use reuse signed in state with fixtures?
2. Is there any example of how to use playwright to test apps with KC authentication?
3. Why isn't the session created when running the global-setup?
4. Why isn't the session being closed while implementing the beforeAll() workaround?

[rwoll]

Thanks for the question and the example code snippets!

One piece that stands out from your original snippet:

  await page.click("text=Sign in");
  // // Save signed-in state to 'storageState.json'.
  const __dirname = path.resolve("playwright/fixtures/storageState.json");
  console.log("Current directory:", __dirname);
  await page
    .context()
    .storageState({ path: "playwright/fixtures/storageState.json" });

This clicks sign in, but then immediately dumps storageState without waiting for a confirmation that the page is signed in (e.g. waiting for an element that appears on the screen like "Hello, miguelofbc!" or a network response). Before dumping storage state, you could try adding an expect that asserts you are actually signed in before saving the session state. (Right now, it might not be working since it might still be in the process of signing in.) Something like:

  await page.click("text=Sign in");
  // // Save signed-in state to 'storageState.json'.
  const __dirname = path.resolve("playwright/fixtures/storageState.json");
  await expect(page.locator("text=Welcome, miguelofbc!")).toBeVisible(); // CHANGE THIS TO WHATEVER IS A SIGNAL OF SUCCESSFUL AUTH IN YOUR CASE
  await page
    .context()
    .storageState({ path: "playwright/fixtures/storageState.json" });

(Or re-use the same helper functions you use in beforeAll since it sounds like you might already being doing some assertions in there if it's working:

  loginPage = new LoginPage(page);
  await loginPage.login(process.env.PW_USERNAME, process.env.PW_PASSWORD);

What does impl of loginPage.login look like?

If the above doesn't help, please additionally answer:

I've been able to understand that the cookie token was not valid when the test was running. After that, I realise that the KC auth session was not active after the global-setup execution.

In this case what does "not valid" mean? How did you determine this or what error are you seeing? Is the KC authentication module you're using strictly cookie-based for authentication or does it track other state?

---

1. Is there any example that I can use as a reference on how to use reuse signed in state with fixtures?

We just added a snippet that covers this (or is at least highly related): https://playwright.dev/docs/next/test-auth#avoiding-multiple-sessions-per-account-at-a-time

2. Is there any example of how to use playwright to test apps with KC authentication?

Not yet!

3. Why isn't the session created when running the global-setup?

See above for a potential fix/explanation).

4. Why isn't the session being closed while implementing the beforeAll() workaround?

Assuming "session" refers to the authentication session, I assume it's because loginPage.login is implemented differently than what's shown in the globalSetup code and the former is either implicitly or explicitly waiting for confirmation of being signed in.

[rwoll]

(Adding assignment and milestone since, if the hypothesis is correct, I want to amend our doc examples to assert a sign-in confirmation.)

[miguelofbc]

@rwoll thanks for the quick follow-up! 🙏

Before dumping storage state, you could try adding an expect that asserts you are actually signed in before saving the session state.

So I've confirmed that the storageState.json was correctly written every time the global-setup script is executed. Can't we conclude that the authentication from global-setup was done properly? Although, already tried this suggestion and had no luck. But just tried again and it failed with:

Global Setup is running...
Current directory: /Users/migueloliveira/development/dev1/worker-safety-client/playwright/fixtures/storageState.json

Error: expect(received).toBeVisible()
Call log:
  - expect.toBeVisible with timeout 5000ms
  - waiting for selector "data-testid=page-layout"

   at ../global-setup.ts:22

  20 |     .storageState({ path: "playwright/fixtures/storageState.json" });
  21 |   // await browser.close();
> 22 |   await expect(page.locator("data-testid=*****")).toBeVisible();
     |                                                         ^
  23 |   await page.close();
  24 |   await context.tracing.stop({ path: "trace.zip" });
  25 |   console.log("Global Setup ended!");

***** = element that should be visible on the home page

So it seems that the navigation to the home page is not happening, but the storageState.json file is being written with the correct cookies. Looking into the trace.zip:

Should global-setup wait for the home page to open? Is that possible? As end-users, the only way I found to debug global-setup is by enabling trace. Are there any logs that would be useful in these circumstances? pw:protocol, pw:browser, pw:api, pw:error? Do u want me to send the logs?

What does impl of loginPage.login look like?

The implementation is as follows:

// login-page.ts
import { Page, expect } from "@playwright/test";

export class LoginPage {
  readonly page: Page;

  constructor(page: Page) {
    this.page = page;
  }

  async login(email: any, password: any) {
    await this.page.locator("input[name=username]").fill(email);
    await this.page.locator("input[name=password]").fill(password);
    await this.page.locator("input", { hasText: "Sign In" }).click();
  }
(...)
}

In this case what does "not valid" mean? How did you determine this or what error are you seeing? Is the KC authentication module you're using strictly cookie-based for authentication or does it track other state?

The token expired as the KC authentication session was already closed. Comment=Expiring cookie As I've mentioned, the KC authentication session is not active when global-setup ends.

We just added a snippet that covers this (or is at least highly related): https://playwright.dev/docs/next/test-auth#avoiding-multiple-sessions-per-account-at-a-time

Thanks! Will take a look! 🙏

[rwoll]

Should global-setup wait for the home page to open? Is that possible?

Yes, the goto you are using should wait: await page.goto(${process.env.BASEURL});. It should behave the same way in global setup as it does in a test/beforeAll.

So I've confirmed that the storageState.json was correctly written every time the global-setup script is executed. Can't we conclude that the authentication from global-setup was done properly?

It might have contents, but that doesn't mean the contents are valid/working. Successfully using it indicates that.

Are you using storageState saving in multiple places at the same time? Perhaps one is overwriting the other.

As end-users, the only way I found to debug global-setup is by enabling trace. Are there any logs that would be useful in these circumstances? pw:protocol, pw:browser, pw:api, pw:error? Do u want me to send the logs?

Traces for global-setup is the best at the moment. Yes, can you send me logs and a trace of your global setup? rosswollman \ m i c r o s o f t . c o m. It would also be helpful if you sent a link to repo that contains the minimal amount of code that shows the problem.

e.g. create a repo, run npm init playwright, and the minimal amount of code possible that shows the problem/failure when npx playwright test is run.

[rwoll]

Btw, it could very well be that KC has a very short sessions, such that a short amount of inactivity causes the session to be expired on the backend. (https://www.keycloak.org/docs/latest/server_admin/index.html#_timeouts) Or that KC, if multiple browsers re-use the same user's cookies/localstorage/auth session, it immediately invalidates that session for everyone. (If that's the case, https://playwright.dev/docs/next/test-auth#avoiding-multiple-sessions-per-account-at-a-time will cover working around it.)

A minimal repro will be a good next step, and then debugging from there about whether the cookie is valid or which part of the auth system considers it invalid and why.

[rwoll]

I attempted to repro here (https://github.com/rwoll/playwright-keycloak-repro), but it looks like I'm able to use KeyCloak session from global-setup within the tests.

Can you try modifying https://github.com/rwoll/playwright-keycloak-repro such that it exhibits the error you're seeing? (I don't have a KC instance, so that PW config starts a local KC container and uses it in the tests. You can comment out the webServer block, and change the URLs to your app.

The repo's README has instructions to run.

[miguelofbc]

@rwoll thanks for the quick reply and for setting up that repo. 🙏 I tried with an empty local repo as well and was hopefully able to understand the problem.

The problem appears to be on the await page.click('text=Sign In'). If we change this line to be await page.locator('input:has-text("Sign In")').click();, as you were suggesting in the playwright-keycloak-repro, everything works as expected.

Actual: ❌ await page.click('text=Sign in');

Expected: ✅ await page.locator('input:has-text("Sign In")').click();

I also forked the playwright-keycloak-repro repo to make sure I was able to reproduce the failure there, so u can further debug if u feel like that's important. You just need to clone https://github.com/miguelofbc/playwright-keycloak-repro and run from there in order to get the repro.

I've also tried this new code example you recently added to the docs and confirmed that the click issue is also there https://playwright.dev/docs/test-advanced#capturing-trace-of-failures-during-global-setup. Wondering if there is no other way to make sure that the global-setup is working as expected.

If you are able to confirm the failure on your end, considering the current page.click() implementation, I would suggest update the docs at least under the links as follows, replacing await page.click('text=Sign in'); with await page.locator('input:has-text("Sign In")').click();.

1. https://playwright.dev/docs/test-auth#sign-in-with-beforeeach
2. https://playwright.dev/docs/test-auth#reuse-signed-in-state
3. https://playwright.dev/docs/test-advanced#global-setup-and-teardown
4. https://playwright.dev/docs/test-advanced#capturing-trace-of-failures-during-global-setup

[rwoll]

@miguelofbc Thanks! I ran your changes, and here's what's going on:

This timing is very nice :) We just updated all the docs (https://github.com/microsoft/playwright/pull/15586, https://github.com/microsoft/playwright/pull/15603) on next to use Locator notation which is preferred over using click (or fill, etc.) directly off the page. Locators are strict by default to prevent accidentally clicking on the wrong thing in the case of ambiguity. Sorry this hadn't been updated sooner!

await page.click('text=Sign in'); // 👎 non-strict, will just click on the first match
await page.locator('text=Sign in').click(); // 👍 strict, will throw an error if there's not exactly one match

If I use the Locator from above with KC page, we see the following error:

locator.click: Error: strict mode violation: "text=Sign In" resolved to 2 elements:
    1) <h1 id="kc-page-title">        Sign in to your account↵↵</h1> aka playwright.$("text=Sign in to your account")
    2) <input tabindex="4" name="login" id="kc-login" type="su…/> aka playwright.$("input:has-text("Sign In")")

So, in the broken 👎 case Playwright was just clicking on the \<h1>. In this particular case, adding the input qualifier removes the ambiguity and ensures the intended element is clicked. 👍 You could also write it as await page.locator('input', { hasText: 'Sign In' }).

Cheers! 🎉