Storyshots is unable to run tests using Angular 12 or 13 with storybook suite 6.4.9 ot 6.5.0-alpha.4

DaveMBush commented 2 years ago

Describe the bug Storyshots is not running tests using Angular 12 or 13 with storybook and addons 6.4.9

To Reproduce I am using storybook with Nrwl/NX in order to get the Jest testing to work easily. I tried using the repro wizard but that only produced an angular/cli project and did not get me to a point where I could run the storyshots so, I'm supplying the exact steps to produce a minimal project that demonstrates my issue along with a link to a repo with the results of these steps.

The following will recreate my issue:

Steps to reproduce storyshot error using Angular ~13.0.0 and storybook ~6.4.5

Create new NX Workspace with Angular

npx create-nx-workspace --preset=angular

Respond to the questions:

workspace=demo-workspace
application=demo
CSS
no Cloud
Add Storybook plugin to NX

npm i -D @nrwl/storybook

Add Storybook nrwl schematic

npm i -D @nrwl/storybook

Add Storybook to demo app

npx nx g @nrwl/angular:storybook-configuration demo

Answer the questions:

cypress=n
Auto generate stories=y
Auto generate specs=n
There is a known issue that requires us to rollback the angular version from ~13.1.0 to ~13.0.0. Do so and run npm install again.
At this point you should be able to run storybook

npx nx run demo:storybook

Add storyshots to the project

npm install @storybook/addon-storyshots

Add storyshots.spec.js to apps/demo/src and put the standard two lines in the file:

import initStoryShots from '@storybook/addon-storyshots';

initStoryShots();

Add the following sections to jest.config.js file in /apps/demo

  moduleFileExtensions: ['ts', 'js', 'html', 'json', 'mjs'],
  moduleNameMapper: {
    'jest-preset-angular/build/setup-jest': 'jest-preset-angular/setup-jest',
    'jest-preset-angular/build/AngularNoNgAttributesSnapshotSerializer':
      'jest-preset-angular/build/serializers/no-ng-attributes',
    'jest-preset-angular/build/AngularSnapshotSerializer':
      'jest-preset-angular/build/serializers/ng-snapshot',
    'jest-preset-angular/build/HTMLCommentSerializer':
      'jest-preset-angular/build/serializers/html-comment'
  },

This solves the spdx* issue that you would otherwise see and maps functions that no longer exists to where they've been moved to.

At this point, you should be able to run the tests and see the error

npm run test

Errors on storyshots.spec.js with: "Your test suite must contain at least one test."

A fully setup project can be found https://github.com/DaveMBush/demo-workspace

System Please paste the results of npx sb@next info here. Environment Info:

System: OS: Windows 10 10.0.22000 CPU: (16) x64 AMD Ryzen 7 5800X 8-Core Processor Binaries: Node: 16.13.1 - C:\Program Files\nodejs\node.EXE Yarn: 3.1.1 - ~\AppData\Roaming\npm\yarn.CMD npm: 8.3.0 - C:\Program Files\nodejs\npm.CMD Browsers: Chrome: 96.0.4664.110 Edge: Spartan (44.22000.120.0), Chromium (96.0.1054.62) @storybook/manager-webpack5: ~6.4.5 => 6.4.9

Additional context I get the same result if I upgrade the storybook packages to 6.5.0-alpha.4

I've added console.log messages to the code and I can see that globalWindow.__STORYBOOK_STORY_STORE__.initializationPromise.then() never resolves (ie, then() never gets called) and storybook.raw() returns an empty array. My only conclusion is that something else needs to get called before my call to testStorySnapshots() but I am not able to determine what that might be.

SethDavenport commented 2 years ago

seeing the same thing with a React project, sb 6.4.9. In my case I am not using NX

SethDavenport commented 2 years ago

Searching the code for initializationPromise yields no other hits, so I'm not sure where this variable gets set.

SethDavenport commented 2 years ago

Looks from searching the code that something should be calling globalWindow.__STORYBOOK_STORY_STORE__.resolveInitializationPromise, but a quick code search turns up no actual calls to this function: https://github.com/storybookjs/storybook/search?q=resolveInitializationPromise

dbush-clickup commented 2 years ago

@SethDavenport I'm convinced now that there is some interaction between Jest and Storybook that isn't happening and that initialization code is something that Jest calls, but isn't in this case.

SethDavenport commented 2 years ago

Yeah that's a possibility - I guess the other thing that changed in the given timeframe is we upgraded from jest 26 to jest 27. However our other (non-monorepo) repos seem to be working with Jest 27.

I should note that the problematic repo is a monorepo with yarn workspaces; you mentioned nx - maybe it's something to do with how deps are managed in monorepos?

DaveMBush commented 2 years ago

Yes, I believe monorepos is part of the issue here.

DaveMBush commented 2 years ago

BTW, globalWindow.STORYBOOK_STORY_STORE is something that gets initialized in Storybook and used in Storyshots. That much I've been able to sluth out.

Also the initializationPromise never hits the resolve state when run from the workspace which is why jest doesn't see any tests

jamesjara commented 2 years ago

Same issue using react-native FYI

resource11 commented 2 years ago

Same deal happens when trying to upgrade to Storybook 6.4.13 from 6.3.x for React 16.13.1

dbush-clickup commented 2 years ago

@resource11 Are you using Jest 27?

dannyshaw commented 2 years ago

Same issue. Jest 27.4.7 Storybook 6.4.13 React 16.14.0

resource11 commented 2 years ago

@dbush-clickup, no I'm not. And I'm not using NX

    "jest": "26.1.0",
    "jest-axe": "^5.0.1",
    "jest-cli": "26.1.0",
    "jest-environment-jsdom": "27.0.3",
    "jest-environment-jsdom-global": "2.0.4",
    "jest-junit": "5.2.0",
    "jest-localstorage-mock": "2.4.2",

g1eny0ung commented 2 years ago

Same issue. Jest 27.4.7 Storybook 6.4.13 React 16.14.0

I don't guarantee this is related to your problem, after I upgraded to the same dependency as you, I found that storyshots test doesn't work anymore because jest changed the default testEnvironment to node, manually changing it to jsdom allows to use the browser-like environment.

Example:

"jest": {
  "testEnvironment": "jsdom"
}

Ref: https://jestjs.io/blog/2021/05/25/jest-27.

DaveMBush commented 2 years ago

@g1eny0ung Thanks for the information. Nrwl sets that for me. I just double-checked.

BruceWeng commented 2 years ago

My issue looks like this:

➜ npx jest storyshots.test.js
ngcc-jest-processor: running ngcc 
FAIL  src/stories/storyshots.test.js  

    ● Test suite failed to run

      Cannot set base providers because it has already been called

      16 |
      17 | import initStoryshots from '@storybook/addon-storyshots';
    > 18 | initStoryshots();
         |               ^
      19 |

I suspect it's because I have init test env in the setup-jest.ts

//setup-jest.ts
import { getTestBed } from '@angular/core/testing';
import { BrowserDynamicTestingModule, platformBrowserDynamicTesting } from '@angular/platform-browser-dynamic/testing';

getTestBed().initTestEnvironment(BrowserDynamicTestingModule, platformBrowserDynamicTesting());

However, the setup above is what we are currently using in Angular 13. Is there anyway I can resolve this error?

storybook: 6.5.0-alpha.23. jest: 27.4.7. angular: 13.1.1.

tomasfrancisco commented 2 years ago

I have the exact same issue, although my test runner only displays:

 FAIL  src/stories.spec.tsx
  ● Test suite failed to run

    Your test suite must contain at least one test.

      at onResult (node_modules/@jest/core/build/TestScheduler.js:175:18)
      at node_modules/@jest/core/build/TestScheduler.js:316:17
      at node_modules/emittery/index.js:260:13
          at Array.map (<anonymous>)
      at Emittery.emit (node_modules/emittery/index.js:258:23)

Also, I've run the same debugging process described already on this thread, and the same promise globalWindow.__STORYBOOK_STORY_STORE__.initializationPromise never gets resolved.

Here's a reproduction repo/commit: https://github.com/tomasfrancisco/react-component-library-template/tree/10ab634f0a55ea97e3e988a7690095e3d2ec95c1

macrozone commented 2 years ago

happend to me as well with react, when upgrading storybook from 6.4.0 to 6.4.18

dbush-clickup commented 2 years ago

happend to me as well with react, when upgrading storybook from 6.4.0 to 6.4.18

Interesting. so, it worked on 6.4.0 but sometime since then it broke?

macrozone commented 2 years ago

happend to me as well with react, when upgrading storybook from 6.4.0 to 6.4.18

Interesting. so, it worked on 6.4.0 but sometime since then it broke?

i try now to bisect to find the version that breaks it

macrozone commented 2 years ago

could not find the issue, i think its not directly related to storybook, but i have no idea what causes the issue.

senanmsft commented 2 years ago

Replacing export default with module.exports in .storybook/main.ts resolves the issue.

PR on @tomasfrancisco's repo

DaveMBush commented 2 years ago

Replacing export default with module.exports in .storybook/main.ts resolves the issue.

PR on @tomasfrancisco's repo

My code is already using module.exports 🤷‍♂️

We changed our workspace for the mono-repo to just run this set of tests manually for now. Not a great fix but it moves us along.

senanmsft commented 2 years ago

Apologies, I missed that your repo was a different issue: you were missing a regex for storybook to find your stories PR

DaveMBush commented 2 years ago

Apologies, I missed that your repo was a different issue: you were missing a regex for storybook to find your stories PR

Because it is specified in the demo app. This is a monorepo issue. And, according to all the documentation I've read and how this has worked in the past, the way I have things setup is correct.

syntactic-salt commented 2 years ago

Has anyone figured out a workaround for this issue?

dbush-clickup commented 2 years ago

Has anyone figured out a workaround for this issue?

We were able to create a manual test runner for the project that changes the working directory into the project prior to running the tests.

In NX it looks like:

    "test": {
      "executor": "@nrwl/workspace:run-commands",
      "outputs": ["coverage/libs/core-components"],
      "options": {
        "cwd": "libs/core-components",
        "commands": ["npx jest --config=jest.config.js"]
      }
    },

scottdickerson commented 2 years ago

This is still broken in the latest versions of both jest and storybook

        "@storybook/addon-storyshots": "6.5.4",
        "jest": "^28.1.0",

Summary of all failing tests
 FAIL  .storybook/storyshots.test.ts
  ● Test suite failed to run

    Your test suite must contain at least one test.

sellmeadog commented 2 years ago

I was able to fix all of my Nx + Storybook + Storyshots issues with this configuration:

import { render } from '@testing-library/react';
import { resolve } from 'path';
import initStoryshots from '@storybook/addon-storyshots';

initStoryshots({
  configPath: resolve(__dirname, '../.storybook'),
  framework: 'react',
  renderer: render,
});

The crux of the issue seems to be that by default, storybook/storyshots cannot find the .storybook directory specific to the app/lib being tested which must be provided with the configPath option. Once that was set, all of the stories were discovered and I subsequently had to configure framework and renderer to resolve some runtime issues and then everything started working as expected.

ianwremmel commented 2 years ago

For what it's worth, none of the solutions suggested in this issue have worked for me. Everything was fine until this change:

diff --git a/package.json b/package.json
index 1e4d2ad..f72c103 100644
--- a/package.json
+++ b/package.json
@@ -69,10 +69,10 @@
     "@storybook/manager-webpack5": "~6.5",
     "@storybook/preset-scss": "~1.0.3",
     "@storybook/react": "~6.5.9",
-    "@testing-library/jest-dom": "^5.11.4",
+    "@testing-library/jest-dom": "^5.16.4",
     "@testing-library/react": "^12.1.2",
     "@types/dedent": "^0.7.0",
-    "@types/jest": "^26.0.14",
+    "@types/jest": "^28.1.6",
     "@types/luxon": "^3.0.0",
     "@types/react": "^17.0.13",
     "@types/react-dom": "^17.0.8",
@@ -93,8 +93,9 @@
     "eslint-plugin-react-hooks": "^4.6.0",
     "husky": "^8.0.1",
     "identity-obj-proxy": "^3.0.0",
-    "jest": "^26.4.2",
-    "jest-junit": "^12.0.0",
+    "jest": "^28.1.3",
+    "jest-environment-jsdom": "^28.1.3",
+    "jest-junit": "^14.0.0",
     "lint-staged": "^13.0.3",
     "markdown-toc": "^1.2.0",
     "mkdirp": "^1.0.4",

Files of interest:

  "dependencies": {
    "bootstrap": "^5.2.0",
    "classnames": "^2.2.6",
    "dedent": "^0.7.0",
    "lodash": "^4.17.21",
    "luxon": "^3.0.1",
    "react": "^17.0.2",
    "react-bootstrap": "^2.4.0",
    "react-dom": "^17.0.2"
  },
  "devDependencies": {
    "@babel/cli": "^7.18.9",
    "@babel/core": "^7.18.9",
    "@babel/preset-env": "^7.18.9",
    "@babel/preset-react": "^7.18.6",
    "@babel/preset-typescript": "^7.18.6",
    "@commitlint/cli": "^12.0.1",
    "@commitlint/config-conventional": "^12.0.1",
    "@faker-js/faker": "^7.3.0",
    "@ianwremmel/eslint-plugin-ianwremmel": "^6.2.0",
    "@storybook/addon-actions": "~6.5.9",
    "@storybook/addon-essentials": "~6.5.9",
    "@storybook/addon-links": "~6.5.9",
    "@storybook/addon-storyshots": "~6.5.9",
    "@storybook/builder-webpack5": "~6.5",
    "@storybook/manager-webpack5": "~6.5",
    "@storybook/preset-scss": "~1.0.3",
    "@storybook/react": "~6.5.9",
    "@testing-library/jest-dom": "^5.16.4",
    "@testing-library/react": "^12.1.2",
    "@types/dedent": "^0.7.0",
    "@types/jest": "^28.1.6",
    "@types/luxon": "^3.0.0",
    "@types/react": "^17.0.13",
    "@types/react-dom": "^17.0.8",
    "@typescript-eslint/eslint-plugin": "^5.31.0",
    "@typescript-eslint/parser": "^5.31.0",
    "babel-loader": "^8.2.5",
    "chromatic": "^6.7.2",
    "css-loader": "^5.0.0",
    "eslint": "^8.20.0",
    "eslint-config-prettier": "^8.5.0",
    "eslint-plugin-babel": "^5.3.1",
    "eslint-plugin-eslint-comments": "^3.2.0",
    "eslint-plugin-import": "^2.26.0",
    "eslint-plugin-jsx-a11y": "^6.6.1",
    "eslint-plugin-markdown": "^3.0.0",
    "eslint-plugin-prettier": "^4.2.1",
    "eslint-plugin-react": "^7.30.1",
    "eslint-plugin-react-hooks": "^4.6.0",
    "husky": "^8.0.1",
    "identity-obj-proxy": "^3.0.0",
    "jest": "^28.1.3",
    "jest-environment-jsdom": "^28.1.3",
    "jest-junit": "^14.0.0",
    "lint-staged": "^13.0.3",
    "markdown-toc": "^1.2.0",
    "mkdirp": "^1.0.4",
    "npm-run-all": "^4.1.5",
    "prettier": "^2.7.1",
    "react-test-renderer": "^17.0.2",
    "rimraf": "^3.0.2",
    "sass": "^1.26.11",
    "sass-loader": "^10.0.2",
    "semantic-release": "^17.1.2",
    "style-loader": "^2.0.0",
    "typescript": "^4.7.4",
    "webpack": "^5.74.0"
  },

/** @file main.js */
module.exports = {
  core: {
    builder: 'webpack5',
  },
  "stories": [
    "../src/**/*.stories.mdx",
    "../src/**/*.stories.@(js|jsx|ts|tsx)"
  ],
  "addons": [
    '@storybook/preset-scss',
    "@storybook/addon-links",
    "@storybook/addon-essentials"
  ],
  typescript: {
    // We check typescript elsewhere; this should speed up builds
    check: false
  }
}

/** @file preview.js */
import React from 'react';
import {faker} from '@faker-js/faker';

export const decorators = [
  (Story) => {
    faker.seed(1701);
    return <Story />;
  },
];

export const parameters = {
  actions: {argTypesRegex: '^on[A-Z].*'},
};

/** @file storyshots.spec.ts */
import initStoryshots, {
  multiSnapshotWithOptions,
} from '@storybook/addon-storyshots';

initStoryshots({
  test: multiSnapshotWithOptions(),
});

/** @files jest.config.js */
'use strict';

const CI = !!process.env.CI;

module.exports = {
  clearMocks: true,
  collectCoverage: false,
  coverageDirectory: 'reports/coverage',
  coverageProvider: 'v8',
  moduleNameMapper: {
    '\\.(css|scss)$': 'identity-obj-proxy',
    '\\.(jpg|jpeg|png|gif|eot|otf|webp|svg|ttf|woff|woff2|mp4|webm|wav|mp3|m4a|aac|oga)$':
      '<rootDir>/__mocks__/file-mock.js',
  },
  reporters: [
    'default',
    CI && [
      'jest-junit',
      {
        ancestorSeparator: ' › ',
        classNameTemplate: '{classname}',
        includeConsoleOutput: true,
        outputDirectory: 'reports/junit',
        outputName: `jest${
          process.env.BUILDKITE_JOB_ID ? `+${process.env.BUILDKITE_JOB_ID}` : ''
        }.xml`,
        suiteName: 'Unit Tests',
        titleTemplate: '{title}',
      },
    ],
  ].filter(Boolean),
  setupFilesAfterEnv: ['<rootDir>/jest.d/setup-files-after-env/faker.ts'],
  testEnvironment: 'jsdom',
  testMatch: [
    '**/src/**/__tests__/**/*.[jt]s?(x)',
    '**/src/**/?(*.)+(spec|test).[tj]s?(x)',
  ],
  testPathIgnorePatterns: ['/node_modules/'],
  transform: {
    '^.+\\.stories\\.[tj]sx?$': '@storybook/addon-storyshots/injectFileName',
    // babel-jest is loaded automatically when transform is null
    // eslint-disable-next-line sort-keys
    '^.+\\.[tj]sx?$': 'babel-jest',
  },
};

/** @file babel.config.js */
'use strict';

module.exports = function (api) {
  api.cache.using(
    () =>
      `${process.env.BABEL_ENV}${process.env.NODE_ENV}${process.env.BUILD_TARGET}`
  );

  const config = {
    comments: true,
    presets: [
      '@babel/preset-typescript',
      [
        '@babel/preset-react',
        {
          development:
            process.env.BABEL_ENV === 'development' ||
            process.env.NODE_ENV === 'development',
        },
      ],
      [
        '@babel/preset-env',
        {
          modules: process.env.BUILD_TARGET === 'modules' ? false : undefined,
          targets: process.env.BUILD_TARGET
            ? 'last 2 versions, > 5%, not dead'
            : {node: true},
        },
      ],
    ],
    retainLines: process.env.NODE_ENV !== 'production',
    sourceMaps: true,
  };

  return config;
};

ndelangen commented 1 year ago

The future of storyshots is the test-runner: https://storybook.js.org/docs/react/writing-tests/test-runner#page-top

And use the play function for expectations: https://storybook.js.org/docs/react/writing-stories/play-function#page-top

We will not be making any improvement / changes to storyshots.

ianwremmel commented 1 year ago

@ndelangen Can you expand on that? Looking at the docs for the test-runner, it appears that snapshot testing is something explicitly different. The docs days that the test runner will ensure stories render, but it doesn't say anything about ensuring they render consistently. Will there be a straightforward way to snapshot every story or will it be necessary to right a play function for every story that manually calls a snapshot expectation?

Last time I tried the test runner, it was kinda painful to get running because its cli doesn't really behave like most other test runners (I don't remember the details anymore). Are there plans to integrate it into Jest (like Storyshots) or into the new built-in Node test runner?

ndelangen commented 1 year ago

@ianwremmel sure, we have hooks for the test-runner defined here: https://storybook.js.org/docs/react/writing-tests/test-runner#advanced-configuration

These allow you to run some code on the page after the story rendered, like taking real screenshots, or snapshotting the DOM.

ianwremmel commented 1 year ago

Got it, thanks! Any chance there'll be a migration guide for how to replace storyshots hooks? It's not obvious from the hooks docs that you can use them to create a "test".

ndelangen commented 1 year ago

@ianwremmel thanks for your continued questions!

It seems what you want to to integrate your stories into your existing tests?

@yannbf created this package a little while ago: https://www.npmjs.com/package/@storybook/testing-react

Now that's react-specific (might fit your needs still), but our goal is to make this available in all storybook renderers. We've already moved some code over: https://github.com/storybookjs/storybook/blob/43f93bb8fb8056fbbd46f6d5fd90ca460c5f858a/code/renderers/react/src/testing-api.ts#L98-L134

I don't think it supports other renderers, right now? @yannbf We'd love some help getting that into shape!

What has been clear to the storybook team for some time, is that the default setup of storyshots (taking jest snapshots of stories) is more of an anti-pattern rather then a useful tool, and besides it not being as useful because of that, it's also been a maintenance chore for the storybook team. The test-runner leverages playwright to run stories in an actual browser.

The testing utils are designed to make stories act as the setup phase of any unit test.

I hope this answer helps you. 🙏

ianwremmel commented 1 year ago

It seems what you want to to integrate your stories into your existing tests?

@ndelangen Kinda? I'd like to keep the current behavior: I write a bunch of stories that are automatically snapshotted without having to write discrete expectations for each story.

more of an anti-pattern rather then a useful tool

Can you expand on that? Is the reasoning documented somewhere? It's definitely been clear that storyshots doesn't get the same level of maintenance as the rest of storybook, but it's been a hugely helpful tool for me to catch unexpected changes.

ndelangen commented 1 year ago

The issue for us is that nearly all of storybook's code is supposed to run in the browser. Storyshots needs to run in node.

We have been trying real hard to improve performance. The implementation of storyshots has been getting in the way many times.

We've build the test-runner which is extensible. You could extract the html and store it somewhere and re-create most of the experience of storyshots, but it's definitely not as easy and as feature rich as storyshots is.

I have personally found that taking snapshots of stories outputted JSX or html does catches changes, but it doesn't catch all. It gives confidence, but this confidence is easily broken. Stylings changes don't show up, and small html changes that don't matter to the actual user interface show up.

Updating snapshots is a breeze, but this is a blessing and a curse. Who reviews all those changed snapshots?

I think a visual regression tool on top of storybook is a better solution. I'm obviously inclined to recommend chromatic.

ianwremmel commented 1 year ago

I'm obviously inclined to recommend chromatic.

Yea, Chromatic is great if you can afford it, but it's all but unusable unless you're VC-backed or very profitable.

Storyshots needs to run in node.

That's fair, but couldn't playwright take care of this? Doesn't it provide a pretty simple way to capture a DOM subtree? I could definitely be wrong about this, but I thought playwright tests still executed in node, but they sent instructions to a running browser instance.

We've build the test-runner which is extensible.

From what I can tell, the test runner is almost usable for this, but there's no good way to write a "templated test" for lack of a better term. I'd love to be able to write something like

forEachStory(Story) => {
  describe(Story.name, () => {
    it('renders', () => {
      expect(() => <Story/>).not.toThrow();
    });

    it('renders deterministically', () => {
      expect(() => <Story/>).toMatchSnapshot();
    });
  });
});

ianwremmel commented 1 year ago

oh, looks like I was wrong about that last bit. that's exactly what postRender is for.

asinitson commented 1 year ago

there's no good way to write a "templated test" for lack of a better term

This is exactly what I am looking for: just answering is any of the stories broken? Sort of a smoke/sanity check. And all that without writing tonnes of boilerplate (since Storybook knows about the full list of stories, why not just iterate over them?). So even expect(() => <Story/>).not.toThrow(); for every story would already be a nice check (I would even drop the snapshot matching part).

Chromatic and other kind of tools are more granular and are not exactly in conflict with this kind of basic test: they provide more fine grained checks which not all projects are mature enough to commit to.

ndelangen commented 1 year ago

... even expect(() => <Story/>).not.toThrow(); for every story would already be a nice check ...

@asinitson You are describing the default behavior of the test-runner.

storybookjs / storybook

Storyshots is unable to run tests using Angular 12 or 13 with storybook suite 6.4.9 ot 6.5.0-alpha.4 #17047