"storybook init" builds wonky package-lock.json using npm v8

JohnAlbin commented 2 years ago

Describe the bug

When using React v18 and npm v8, the storybook init command command creates an invalid package-lock.json that no one else can use to create a properly-running Storybook installation.

To Reproduce

I tried using npx sb@next repro but that forces the usage of yarn and this is an npm-related bug. Because of deployment system constraints, I have to use npm.

Fortunately, the steps to reproduce are very simple though and I've added the resulting files to a Git repo.

npm init -y to create a package.json in the current directory.
npm install react react-dom installs React 18.1.0 93df36f
npx storybook init -N --builder webpack5 --type react to install Storybook 6.5.3 77d78d0

The resulting installation works fine; npm run storybook builds fine and all default Storybook pages render correctly.

BUT, the package-lock.json cannot be reused (not by yourself when adding more dependencies, nor by a CI system, nor by another dev using a Git clone). If you run npm ci, it will report the following error:

npm ci can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with npm install before continuing.

That means that no one else will be able to use that package-lock.json to get the same set of packages currently installed.

Running npm install changes the package-lock.json file (61bcd0f) but MDX files no longer render correctly (see #18266).

System

Environment Info:

  System:
    OS: macOS 12.4
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  Binaries:
    Node: 18.1.0 - ~/.nvm/versions/node/v18.1.0/bin/node
    Yarn: 1.22.18 - ~/.yarn/bin/yarn
    npm: 8.8.0 - ~/.nvm/versions/node/v18.1.0/bin/npm
  Browsers:
    Chrome: 101.0.4951.64
    Firefox: 97.0.2
  npmGlobalPackages:
    @storybook/cli: 6.5.3

Additional context

The full error message when running npm ci is:

npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or
npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Missing: require-from-string@2.0.2 from lock file
npm ERR! Missing: typescript@4.6.4 from lock file
npm ERR! Invalid: lock file's acorn@7.4.1 does not satisfy acorn@8.7.1
npm ERR! Missing: acorn@7.4.1 from lock file
npm ERR! Invalid: lock file's postcss@7.0.39 does not satisfy postcss@8.4.14
npm ERR! Missing: postcss@7.0.39 from lock file
npm ERR! Missing: postcss@7.0.39 from lock file
npm ERR! Missing: picocolors@1.0.0 from lock file
npm ERR! Missing: postcss@7.0.39 from lock file
npm ERR! Invalid: lock file's webpack@5.72.1 does not satisfy webpack@4.46.0
npm ERR! Missing: html-webpack-plugin@5.5.0 from lock file
npm ERR! Missing: webpack@5.72.1 from lock file
npm ERR! Missing: terser-webpack-plugin@5.3.1 from lock file
npm ERR! Missing: html-webpack-plugin@5.5.0 from lock file
npm ERR! Missing: webpack@5.72.1 from lock file
npm ERR! Missing: terser-webpack-plugin@5.3.1 from lock file
npm ERR! Invalid: lock file's @webassemblyjs/ast@1.11.1 does not satisfy @webassemblyjs/ast@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/wasm-edit@1.11.1 does not satisfy
@webassemblyjs/wasm-edit@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/wasm-parser@1.11.1 does not satisfy
@webassemblyjs/wasm-parser@1.9.0
npm ERR! Invalid: lock file's acorn@8.7.1 does not satisfy acorn@6.4.2
npm ERR! Invalid: lock file's enhanced-resolve@5.9.3 does not satisfy enhanced-resolve@4.5.0
npm ERR! Invalid: lock file's eslint-scope@5.1.1 does not satisfy eslint-scope@4.0.3
npm ERR! Invalid: lock file's loader-runner@4.3.0 does not satisfy loader-runner@2.4.0
npm ERR! Missing: loader-utils@1.4.0 from lock file
npm ERR! Missing: micromatch@3.1.10 from lock file
npm ERR! Missing: mkdirp@0.5.6 from lock file
npm ERR! Missing: schema-utils@1.0.0 from lock file
npm ERR! Invalid: lock file's terser-webpack-plugin@5.3.1 does not satisfy
terser-webpack-plugin@1.4.5
npm ERR! Missing: watchpack@1.7.5 from lock file
npm ERR! Invalid: lock file's webpack-sources@3.2.3 does not satisfy webpack-sources@1.4.3
npm ERR! Invalid: lock file's @webassemblyjs/helper-wasm-bytecode@1.11.1 does not satisfy
@webassemblyjs/helper-wasm-bytecode@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/helper-buffer@1.11.1 does not satisfy
@webassemblyjs/helper-buffer@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/helper-wasm-section@1.11.1 does not satisfy
@webassemblyjs/helper-wasm-section@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/wasm-gen@1.11.1 does not satisfy
@webassemblyjs/wasm-gen@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/wasm-opt@1.11.1 does not satisfy
@webassemblyjs/wasm-opt@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/wast-printer@1.11.1 does not satisfy
@webassemblyjs/wast-printer@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/ieee754@1.11.1 does not satisfy
@webassemblyjs/ieee754@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/leb128@1.11.1 does not satisfy
@webassemblyjs/leb128@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/utf8@1.11.1 does not satisfy @webassemblyjs/utf8@1.9.0
npm ERR! Invalid: lock file's @webassemblyjs/helper-api-error@1.11.1 does not satisfy
@webassemblyjs/helper-api-error@1.9.0
npm ERR! Missing: memory-fs@0.5.0 from lock file
npm ERR! Missing: cacache@12.0.4 from lock file
npm ERR! Missing: is-wsl@1.1.0 from lock file
npm ERR! Missing: schema-utils@1.0.0 from lock file
npm ERR! Invalid: lock file's serialize-javascript@6.0.0 does not satisfy serialize-javascript@4.0.0
npm ERR! Missing: terser@4.8.0 from lock file
npm ERR! Missing: tapable@2.2.1 from lock file
npm ERR! Missing: jest-worker@27.5.1 from lock file
npm ERR! Missing: serialize-javascript@6.0.0 from lock file
npm ERR! Missing: supports-color@8.1.1 from lock file
npm ERR! Missing: has-flag@4.0.0 from lock file
npm ERR! Missing: @webassemblyjs/ast@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/wasm-edit@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/wasm-parser@1.11.1 from lock file
npm ERR! Missing: enhanced-resolve@5.9.3 from lock file
npm ERR! Missing: eslint-scope@5.1.1 from lock file
npm ERR! Missing: loader-runner@4.3.0 from lock file
npm ERR! Missing: webpack-sources@3.2.3 from lock file
npm ERR! Missing: @webassemblyjs/helper-wasm-bytecode@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/helper-api-error@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/helper-buffer@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/helper-wasm-section@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/wasm-gen@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/wasm-opt@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/wast-printer@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/ieee754@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/leb128@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/utf8@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/helper-api-error@1.11.1 from lock file
npm ERR! Missing: estraverse@4.3.0 from lock file
npm ERR! Missing: postcss@7.0.39 from lock file
npm ERR! Missing: tapable@2.2.1 from lock file
npm ERR! Missing: jest-worker@27.5.1 from lock file
npm ERR! Missing: serialize-javascript@6.0.0 from lock file
npm ERR! Missing: supports-color@8.1.1 from lock file
npm ERR! Missing: @webassemblyjs/ast@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/wasm-edit@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/wasm-parser@1.11.1 from lock file
npm ERR! Missing: enhanced-resolve@5.9.3 from lock file
npm ERR! Missing: eslint-scope@5.1.1 from lock file
npm ERR! Missing: loader-runner@4.3.0 from lock file
npm ERR! Missing: webpack-sources@3.2.3 from lock file
npm ERR! Missing: @webassemblyjs/helper-wasm-bytecode@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/helper-buffer@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/helper-wasm-section@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/wasm-gen@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/wasm-opt@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/wast-printer@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/ieee754@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/leb128@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/utf8@1.11.1 from lock file
npm ERR! Missing: @webassemblyjs/helper-api-error@1.11.1 from lock file
npm ERR! Missing: estraverse@4.3.0 from lock file
npm ERR! Missing: readable-stream@2.3.7 from lock file
npm ERR! Missing: isarray@1.0.0 from lock file
npm ERR! Missing: string_decoder@1.1.1 from lock file
npm ERR! Missing: chownr@1.1.4 from lock file
npm ERR! Missing: lru-cache@5.1.1 from lock file
npm ERR! Missing: mkdirp@0.5.6 from lock file
npm ERR! Missing: rimraf@2.7.1 from lock file
npm ERR! Missing: ssri@6.0.2 from lock file
npm ERR! Missing: y18n@4.0.3 from lock file
npm ERR! Missing: yallist@3.1.1 from lock file
npm ERR! Missing: commander@2.20.3 from lock file
npm ERR! Missing: json5@1.0.1 from lock file
npm ERR! Missing: braces@2.3.2 from lock file
npm ERR! Missing: extend-shallow@2.0.1 from lock file
npm ERR! Missing: fill-range@4.0.0 from lock file
npm ERR! Missing: extend-shallow@2.0.1 from lock file
npm ERR! Missing: is-number@3.0.0 from lock file
npm ERR! Missing: to-regex-range@2.1.1 from lock file
npm ERR! Missing: kind-of@3.2.2 from lock file
npm ERR! Missing: is-extendable@0.1.1 from lock file
npm ERR! Missing: is-buffer@1.1.6 from lock file

shilman commented 2 years ago

Thanks for the reproduction @JohnAlbin -- I can confirm that the repro succeeds with npm6 and fails with npm8.

shilman commented 2 years ago

@merceyz is this something you can take a look at?

gaetanmaisse commented 2 years ago

It looks like it's working with Node 16 + npm 8.5.5, so maybe related to npm 8.5+ 🤔

Update:

✅ 8.5.5
💥 8.6

shilman commented 2 years ago

@JohnAlbin Here's the deal. h/t @ndelangen @yannbf for debugging with me:

npm8 changes the semantics of peer dependencies and Storybook 6 is stuck on an old peer dependency structure (we will fix with a breaking change in SB7)
because of this limitation, sb init detects NPM7+ and runs with --legacy-peer-deps. This is the typical workaround that you'll see in lots of issues in the SB repo
therefore, when you run npm install in your last step, you are counteracting that and it will update the lockfile. if, instead, you run npm install --legacy-peer-deps, nothing will change in the lockfile

If you run without --legacy-peer-deps, you will hit against a different error about makeSerializable, and you can resolve that in npm8 by forcing wepback5 everywhere:

"overrides": {
  "webpack": "^5"
}

JohnAlbin commented 2 years ago

@shilman Thanks, Michael! That is some excellent debugging with your team.

I've confirmed that the flag you mentioned is also needed when you run npm's ci command (not just when running npm install):

$ npm ci --legacy-peer-deps

Without the flag, npm refuses to install anything. With the flag, it builds a proper project with a working Storybook instance.

It seems to me that the package-lock.json should already know how peer dependencies were handled when the package-lock.json file was created and shouldn't require the extra --legacy-peer-deps flag, but that is definitely a limitation of npm and not a problem with Storybook per se.

I found this in the npm ci docs:

NOTE: If you create your package-lock.json file by running npm install with flags that can affect the shape of your dependency tree, such as --legacy-peer-deps or --install-links, you must provide the same flags to npm ci or you are likely to encounter errors. An easy way to do this is to run, for example, npm config set legacy-peer-deps=true --location=project and commit the .npmrc file to your repo.

I tested creating the .npmrc as noted above in my test repo and it worked great.

With the .npmrc, both npm ci and npm i worked without error and without updating the package-lock.json.

# .npmrc
# Needed to install Storybook before Storybook v7.
legacy-peer-deps=true

Is this something that storybook init should be doing to prevent confusion/problems for npm users? If not, we should document npm's wonkiness.

shilman commented 2 years ago

Thanks @JohnAlbin and thanks for the super clean repro. Can the .npmrc be located in the user's project directory? You're right that we should document all this better. I think this could be a nice automigration which explains to the user what is going on and asks if it's OK to create an npmrc. WDYT?

JohnAlbin commented 2 years ago

Can the .npmrc be located in the user's project directory?

Yes, it needs to be created per-project since the peer-deps setting is a per-project decision.

When you run npm config set legacy-peer-deps=true --location=project it automatically creates a .npmrc file next to the project's package.json file.

shilman commented 2 years ago

Hurrah!! I just released https://github.com/storybookjs/storybook/releases/tag/v7.0.0-alpha.5 containing PR #18510 that references this issue. Upgrade today to the @future NPM tag to try it out!

npx sb upgrade --prerelease

Closing this issue. Please re-open if you think there's still more to do.

shilman commented 2 years ago

Yay!! I just released https://github.com/storybookjs/storybook/releases/tag/v7.0.0-alpha.7 containing PR #18522 that references this issue. Upgrade today to the @future NPM tag to try it out!

npx sb upgrade --prerelease

shilman commented 2 years ago

Egads!! I just released https://github.com/storybookjs/storybook/releases/tag/v6.5.10-alpha.0 containing PR #18522 that references this issue. Upgrade today to the @prerelease NPM tag to try it out!

npx sb upgrade --prerelease

shilman commented 2 years ago

Good golly!! I just released https://github.com/storybookjs/storybook/releases/tag/v6.5.10 containing PR #18522 that references this issue. Upgrade today to the @latest NPM tag to try it out!

npx sb upgrade

tv42 commented 2 years ago

Wouldn't it be better to keep the bug open until the workaround is no longer needed? The changes adding .npmrc are a workaround not a fix, and force projects using storybook to accept a behavioral change just for the benefit of storybook.

f3oall commented 2 years ago

This is not a fix, it's just a workaround! Don't close this issue.

pushred commented 2 years ago

Reporting that the workaround didn't get me too far in the real world. Details and reproduction submitted in https://github.com/storybookjs/storybook/issues/19218

Ariane-B commented 2 years ago

The workaround doesn't work for me. It causes @ag-grid-community/vue to be unable to find its peer dependency vue-property-decorator.

Here's what I've tried. The result is the same in every case.

Delete node_modules and use --no-cache for every `npm install
Use npx storybook init, say yes to the "npm 7 migration" thing
Use npm storybook init, say yes to the "npm 7 migration" thing, then delete .npmrc
Install the and add the package.json commands manually.

I don't get any errors during npm install, but Vue CLI fails to compile with this error:

ERROR Failed to compile with 1 error
This dependency was not found:

vue-property-decorator in ./node_modules/@ag-grid-community/vue/lib/AgGridVue.js

To install it, you can run: npm install --save vue-property-decorator Error from chokidar (C:): Error: EBUSY: resource busy or locked, lstat 'C:\hiberfil.sys'

Versions:

vue 2.6.14
@vue/cli-service 4.5.12
@ag-grid-community/client-side-row-model 28.2.0
@ag-grid-community/core 28.2.0
@ag-grid-community/styles 28.2.0
@ag-grid-community/vue 28.2.0
@storybook/addon-actions 6.5.12
@storybook/addon-essentials 6.5.12
@storybook/addon-interactions 6.5.12
@storybook/addon-links 6.5.12
@storybook/builder-webpack4 6.5.12
@storybook/manager-webpack4 6.5.12
@storybook/testing-library 0.0.13
@storybook/vue 6.5.12

Honestly I'm a bit confused as to why you'd be so insistent on using a non-standard peer dependency scheme that you'd spend time to add in a workaround instead of just doing like everyone else. Also confused by how you're committing node_modules unlike seemingly everyone else.

Please standardize your stuff. It's ridiculous that I can't generate a doc because the doc generator has a disagreement with one of my third-party packages about how peer dependencies should work.

Ariane-B commented 2 years ago

Update: Turns out, the two packages seem to actually agree if you have the latest version of npm. I had updated my Node, but due to this problem, I still had an old version of npm to go with it.

I used npm uninstall -g npm to go back to the version of npm that came with Node, and now, without the NPM v7 migration, @ag-grid-community/vue DOES find its peer dependency.

I'm sorry I went off on you guys. But it might be good to add an error message (with engine-strict maybe?) that you require a specific NPM version to be compatible with other packages?

menasheh commented 1 year ago

unclear why this is closed

shilman commented 1 year ago

Gadzooks!! I just released https://github.com/storybookjs/storybook/releases/tag/v7.0.0-beta.19 containing PR #20456 that references this issue. Upgrade today to the @next NPM tag to try it out!

npx sb upgrade --prerelease

Closing this issue. Please re-open if you think there's still more to do.

christopherbauer commented 1 year ago

I'm still getting the automigration message after running the command. Is this anticipated? @shilman

ndelangen commented 1 year ago

@christopherbauer can you provide more details on what you've done and what happened? As is, I do not have enough information to know what to do.

Can you supply a reproduction?

christopherbauer commented 1 year ago

@christopherbauer can you provide more details on what you've done and what happened? As is, I do not have enough information to know what to do.

Can you supply a reproduction?

I was just wondering if it should just work or if I would get prompted either way.

shilman commented 1 year ago

@christopherbauer try running

npx sb@next upgrade --prerelease

jorgeguberte commented 1 year ago

@christopherbauer try running
npx sb@next upgrade --prerelease

Worked for me with Vue 3, Node 18.12.1

melMass commented 1 year ago

Shouldn't this be specified in the quickstart? https://storybook.js.org/docs/react/get-started/install

I had to run

npx sb init

edit: for vite it's now in the main monorepo 🚀 https://storybook.js.org/blog/first-class-vite-support-in-storybook/

shilman commented 1 year ago

Cc @jonniebigodes i wonder whether we should specify @next in the 7.0 docs

yannbf commented 1 year ago

Cc @jonniebigodes i wonder whether we should specify @next in the 7.0 docs

same goes for sections like testing, where we tell users to install @storybook/testing-library, @storybook/test-runner and @storybook/jest, which need @next as well

curran commented 1 year ago

I'm also seeing the following message after following instructions in the docs:

│   We've detected you are running npm 8.19.3                                                       │
│    which has peer dependency semantics which Storybook is incompatible with.                      │
│                                                                                                   │
│   In order to work with Storybook's package structure, you'll need to run `npm` with the          │
│   `--legacy-peer-deps=true` flag. We can generate an `.npmrc` which will do that automatically.   │
│                                                                                                   │
│   More info: https://github.com/storybookjs/storybook/issues/18298

Environment:

node --version v19.1.0
npm --version 8.19.3
Ubuntu Linux

Steps to reproduce:

npm create vite@latest my-vue-app -- --template react
npx storybook init

As a user new to Storybook, I'm left feeling confused.

I see several options before me, and I'm not sure which one to choose and why:

npx storybook@next automigrate - No idea what this will do
npx sb@next upgrade --prerelease - Seems sketchy, why would I use a prerelease?
npx sb init - Umm this seems like something that should have happened during npx storybook init?

Any guidance is appreciated. Thanks!

curran commented 1 year ago

Sorry, just getting up to speed here on the release status of v7. Decided that I do want to use the prerelease as v7 looks fantastic.

I ended up reverting all changes from npx storybook init and using the following command to add Storybook to my project:

npx sb@next init

This worked fine and I see No migrations were applicable to your project, which is wonderful!

Faulery commented 1 year ago

Hey, I used npx sb@next init works just fine! But, for some reason I noticed that "react": "^18.2.0", "react-dom": "^18.2.0" got installed for the Angular project. Just wondering why it happened?

ndelangen commented 1 year ago

@Faulery it's because of storybook addon-docs. It needs react.

This addon MUST use the same version of react as the user (when the user is using storybook for react)... therefore we needed to make it a peerDependency.

In the past we added react as a dependency to the angular framework, but there was no real dependency-graph connection between addon-docs and the angular framework. Thus the react package being installed was fragile. And in fact since package managers have gotten more strict about these sort of things, we had to do what was right for all, which was to actually force users to install react, and depend on it in their own package.json.

The entire team dreaded doing this, but it was own only real option.

The other option was to split addon docs into multiple packages, and that would have had very bad affects on it's documentation, backwards compatibility and also importantly: maintenance.

I hope this explains the situation + our reasoning.

jadurani commented 1 year ago

I ended up reverting all changes from npx storybook init and using the following command to add Storybook to my project:
npx sb@next init

Confirming that this solution also worked for me 🚀

Environment:

node --version v18.15.0
npm --version 9.5.0
"next": "13.2.4",
"react": "18.2.0",
"react-dom": "18.2.0",
"typescript": "5.0.2"

JackHowa commented 1 year ago

thanks @jadurani -- works for me

storybookjs / storybook

"storybook init" builds wonky package-lock.json using npm v8 #18298