[bug] preVisit accesses the story store before the index is ready

[sergei-maertens]

Describe the bug

We have the test-runner.js implement with the preVisit to apply the different viewports when running tests, per the README.

After upgrading to Storybook 8, our tests fail in a flaky way, e.g.:

 FAIL  src/components/PostCompletionViews/PostCompletionView.stories.js
  ● Private API / Post completion views  › Generic › smoke-test

    page.evaluate: SB_PREVIEW_API_0011 (StoryStoreAccessedBeforeInitializationError): Cannot access the Story Store until the index is ready.

    It is not recommended to use methods directly on the Story Store anyway, in Storybook 9 we will
    remove access to the store entirely

      at Object.get (http:/127.0.0.1:6006/sb-preview/runtime.js:118:5542)
      at __getContext (<anonymous>:215:54)
      at eval (eval at evaluate (:226:30), <anonymous>:1:29)
      at UtilityScript.evaluate (<anonymous>:228:17)
      at UtilityScript.<anonymous> (<anonymous>:1:44)
      at getStoryContext (node_modules/@storybook/test-runner/dist/index.js:15306:15)
      at getStoryContext (.storybook/test-runner.js:9:27)
      at src/components/PostCompletionViews/PostCompletionView.stories.js:151:64
      at step (src/components/PostCompletionViews/PostCompletionView.stories.js:109:23)
      at Object.next (src/components/PostCompletionViews/PostCompletionView.stories.js:50:20)
      at asyncGeneratorStep (src/components/PostCompletionViews/PostCompletionView.stories.js:4:28)
      at _next (src/components/PostCompletionViews/PostCompletionView.stories.js:22:17)
      at src/components/PostCompletionViews/PostCompletionView.stories.js:27:13
      at src/components/PostCompletionViews/PostCompletionView.stories.js:19:16
      at testFn (src/components/PostCompletionViews/PostCompletionView.stories.js:211:49)
      at Object.<anonymous> (src/components/PostCompletionViews/PostCompletionView.stories.js:224:33)
      at step (src/components/PostCompletionViews/PostCompletionView.stories.js:109:23)
      at Object.next (src/components/PostCompletionViews/PostCompletionView.stories.js:50:20)
      at asyncGeneratorStep (src/components/PostCompletionViews/PostCompletionView.stories.js:4:28)
      at _next (src/components/PostCompletionViews/PostCompletionView.stories.js:22:17)
      at src/components/PostCompletionViews/PostCompletionView.stories.js:27:13
      at Object.<anonymous> (src/components/PostCompletionViews/PostCompletionView.stories.js:19:16)

In the browser, there are no problems, only happens when using the test-runner.

To Reproduce

Unfortunately I don't see a way to reproduce this reliably, but our repository is public: https://github.com/open-formulieren/open-forms-sdk/tree/chore/662-sb-upgrade

# clone repo
npm ci
npm run storybook

and in a different tab/shell:

npm run test:storybook

Expected behavior

Tests only run when the story story is ready

Screenshots n/a

System

Storybook Environment Info:

  System:
    OS: Linux 6.7 Arch Linux
    CPU: (16) x64 AMD Ryzen 7 PRO 4750U with Radeon Graphics
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.10.0 - ~/.nvm/versions/node/v20.10.0/bin/node
    Yarn: 1.22.21 - /usr/bin/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm <----- active
    pnpm: 8.15.2 - /usr/bin/pnpm
  npmPackages:
    @storybook/addon-actions: ^8.0.0 => 8.0.0 
    @storybook/addon-coverage: ^1.0.1 => 1.0.1 
    @storybook/addon-essentials: ^8.0.0 => 8.0.0 
    @storybook/addon-interactions: ^8.0.0 => 8.0.0 
    @storybook/addon-links: ^8.0.0 => 8.0.0 
    @storybook/addon-webpack5-compiler-babel: ^3.0.3 => 3.0.3 
    @storybook/components: ^8.0.0 => 8.0.0 
    @storybook/manager-api: ^8.0.0 => 8.0.0 
    @storybook/preview-api: ^8.0.0 => 8.0.0 
    @storybook/react: ^8.0.0 => 8.0.0 
    @storybook/react-webpack5: ^8.0.0 => 8.0.0 
    @storybook/test: ^8.0.0 => 8.0.0 
    @storybook/test-runner: ^0.17.0 => 0.17.0 
    @storybook/types: ^8.0.0 => 8.0.0 
    msw-storybook-addon: ^1.10.0 => 1.10.0 
    storybook: ^8.0.0 => 8.0.0 
    storybook-addon-remix-react-router: ^3.0.0 => 3.0.0 
    storybook-addon-themes: ^6.1.0 => 6.1.0 
    storybook-react-intl: ^3.0.1 => 3.0.1 

Additional context

• I have an outdated theme addon that needs to be replaced still, no idea if that could be messing things up :thinking:

• The relevant code throwing the error appears to be this: https://github.com/storybookjs/storybook/blob/718ae4ad3352d2bc69f95c0cfab25b4b075e2308/code/lib/preview-api/src/modules/preview-web/Preview.tsx#L106

[sergei-maertens]

Interestingly enough, this doesn't seem to happen with a production build since our CI is passing: https://github.com/open-formulieren/open-forms-sdk/actions/runs/8340149970/job/22823603724?pr=663

[AlmarAubel]

We have the same problem. If I add await waitForPageReady(page); i runs less flakky.

const config: TestRunnerConfig = {
  async preVisit(page, story) {
    await waitForPageReady(page);
    }
}

[ItaiCovver]

We have the same problem but adding waitForPageReady stalls indefinitely on this line

await page.waitForLoadState("networkidle")

when running locally and in our CI flow it does not seem to do anything (get the SB_PREVIEW_API_0011 errors). In both we run identical docker containers

Edit: Figured out the discrepancy. In local storybook is running via storybook dev and in CI its building storybook and running npx http-server storybook-static. Seems linked to this issue

[VizTheWiz]

I'm having the same issue, it happens when running storybook dev with the --ci flag (Without the CI flag, when the playground webpage is opened, this does not occur). However, it seems to resolve if I immediately run test-runner after this fail, but it takes longer than usual (10s as opposed to around 4s). Another consecutive call to test-runner takes the expected duration.

This only seems to happen in one specific instance, which also relies on Apollo GraphQL client mocked provider and is overall more complex. Smaller standalone packages with simple components do not seem to have this issue, and the tests run successfully.

For reference, this is the output from storybook info, using Vite:

Storybook Environment Info:

  System:
    OS: Linux 6.5 Debian GNU/Linux 11 (bullseye) 11 (bullseye)
    CPU: (12) x64 12th Gen Intel(R) Core(TM) i7-1265U
    Shell: 5.1.4 - /bin/bash
  Binaries:
    Node: 18.19.1 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 10.2.4 - /usr/local/bin/npm <----- active
    pnpm: 8.15.4 - /usr/local/share/npm-global/bin/pnpm
  npmPackages:
    @storybook/addon-essentials: ^8.0.5 => 8.0.5 
    @storybook/addon-interactions: ^8.0.5 => 8.0.5 
    @storybook/addon-links: ^8.0.5 => 8.0.5 
    @storybook/addon-onboarding: ^8.0.5 => 8.0.5 
    @storybook/blocks: ^8.0.5 => 8.0.5 
    @storybook/react: ^8.0.5 => 8.0.5 
    @storybook/react-vite: ^8.0.5 => 8.0.5 
    @storybook/test: ^8.0.5 => 8.0.5 
    @storybook/test-runner: ^0.17.0 => 0.17.0 
    eslint-plugin-storybook: ^0.8.0 => 0.8.0 
    storybook: ^8.0.5 => 8.0.5 

[Mil4n0r]

Any updates on this? I am having the exact same problem.

[noranda]

Same here! Same problem.

I'm doing the same thing where I'm using preVisit to set the viewport in some of our tests. In development mode, we see these errors when using test-runner. When building statically, we don't see the errors.

[kdrbek06]

Same issue here in postVisit hook

async postVisit(page, context) {
    await waitForPageReady(page);
    ...
}

[dorsharonfuse]

any update on this? We're getting the same problem, and the tests run flaky because of it.

[shilman]

@yannbf Have you had a chance to take a look at this?

[Jero786]

Any update on this? I encountered the same issue too.

[jorgeortega]

With a very small set of stories, test-storybook --maxWorkers=1 worked for me locally. Haven't tried yet on CI.

[noranda]

Any update on this? 🥺

[dmaier-indeed]

Also encountering this. My code is using the recipe to set the viewport size as in this example from the docs.

Changing to postVisit sort of works, but then there are still a few renders with whatever the previous viewport size was, so it's flaky

[Geloosa]

Encountering the same with vue components, one time the tests pass, the next time this error occurs, and in different stories every time.

[gromaco]

We faced the same issue after upgrading to Storybook v8. We set the viewport size in preVisit. However, in some conditions (on CI env or not as part of a complex docker-compose), tests in the same docker image work fine.

 page.evaluate: SB_PREVIEW_API_0011 (StoryStoreAccessedBeforeInitializationError): Cannot access the Story Store until the index is ready.

    It is not recommended to use methods directly on the Story Store anyway, in Storybook 9 we will
    remove access to the store entirely

      at Object.get (http:/localhost:6006/sb-preview/runtime.js:8396:17)
      at __getContext (<anonymous>:219:54)
      at eval (eval at evaluate (:234:30), <anonymous>:1:29)
      at UtilityScript.evaluate (<anonymous>:236:17)
      at UtilityScript.<anonymous> (<anonymous>:1:44)
      at getStoryContext (node_modules/@storybook/test-runner/dist/index.js:12861:15)
      at preVisit (.storybook/test-runner.ts:43:42)

[VizTheWiz]

I'm wondering if this issue is even going to get addressed, or rather if for storybook 8 is going to completely phase out test-runner in favor of the newly announced vitest plugin:

https://storybook.js.org/blog/storybook-8-3/ https://storybook.js.org/docs/writing-tests/vitest-plugin?ref=storybookblog.ghost.io

[pedrosousa13]

encountering the same in github actions. Locally works fine. We have the latest storybook version (8.3.3)

[shilman]

@VizTheWiz indeed we are putting a lot of work into the new Vitest stuff, but we will continue to maintain the test runner for the foreseeable future since it's used by a lot of teams, does not require Vite, etc. I'll try to get the team to look into this soon.

[yannbf]

Hey everyone! I'd really love to fix this issue, but without a clear reproduction it's tough. I'd really appreciate to get as much context as possible so I can get to the bottom of this! I have never experienced this issue myself unfortunately.

[Cuuki]

Hey @yannbf! Thanks for taking a look here!

I cannot speak for OG, but my team is having flaky tests with the exact same error code. We have not further investigated on our end because of time limitations and since one rerun always seems to fix this for a couple of runs but I had my eyes on this thread in the meantime.

I can unfortunately not give you a reliable reproduction, as this error only happens occasionally for us, but I might be able to help you pinpoint it by providing details of our setup (most of this has been mentioned before):

1. This specific issue only happens when testing on webkit for us, never with chromium or firefox.
2. Checking our CI history, this happens about every 4th to 6th run (not very helpful but still 😅).
3. The issue only happens with the test-runner and not in the Storybook UI.
4. The error itself always originates in the preVisit hook of the test-runner config (see image below), where we set up viewports with the storyContext and injectAxe (see code snippet).
5. The command we use for the CI looks as follows test-storybook -c src --url http://127.0.0.1:6006 --browsers webkit --junit --ci --maxWorkers=1.
6. We are currently running Storybook 8.3.5 in a monorepo setup.

I hope this is helpful, let me know if there is anything else I can provide!

async preVisit(page, context) {
    await waitForPageReady(page);

    const storyContext = await getStoryContext(page, context);

    // Accesses the story's parameters and retrieves the viewport used to render it
    // https://storybook.js.org/docs/writing-tests/test-runner#accessing-story-information-with-the-test-runner
    const storyViewport =
        storyContext.parameters?.viewport?.defaultViewport;

    const viewportName: ViewportBreakpointToken =
        storyViewport in VIEWPORTS ? storyViewport : DEFAULT_VIEWPORT;

    const { width, height } = VIEWPORTS[viewportName].styles;

    // Configures the Playwright page to use the viewport size
    await page.setViewportSize({
        width: parseFloat(width.replace('px', '')),
        height: parseFloat(height.replace('px', '')),
    });

    await injectAxe(page);
},