search

heroku / buildpacks-nodejs

Heroku's Cloud Native Buildpacks for Node.js applications.
BSD 3-Clause "New" or "Revised" License
2 stars 2 forks source link

Default nuxt app does not work out of the box. #712

Closed jose-fully-ported closed 11 months ago

jose-fully-ported commented 1 year ago

There are a few issues here (maybe all related to pnpm somehow) but:

# create a new nuxt app
npx nuxi@latest init test-app

# setup a pnpm lock file
pnpm install

# run the build
# will fail because the `packageManager` isn't set
pack build --builder heroku/builder:22 test-app

# set the packageManager
contents="$(jq '.packageManager = "pnpm@8.10.2"' package.json)"
echo -E "${contents}" > package.json

# run the build
# will fail because it can't find a dependency
pack build --builder heroku/builder:22 test-app

Build error:

 ERROR  Cannot find package '@vue/compiler-sfc' imported from /layers/heroku_nodejs-pnpm-install/virtual/unplugin-vue-router@0.7.0_vue-router@4.2.5_vue@3.3.7/node_modules/unplugin-vue-router/dist/chunk-QVOU6AJO.mjs

  at new NodeError (node:internal/errors:406:5)
  at packageResolve (node:internal/modules/esm/resolve:789:9)
  at moduleResolve (node:internal/modules/esm/resolve:838:20)
  at defaultResolve (node:internal/modules/esm/resolve:1043:11)
  at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12)
  at ModuleLoader.resolve (node:internal/modules/esm/loader:352:25)
  at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:228:38)
  at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)
  at link (node:internal/modules/esm/module_job:84:36)

 ERROR  Cannot find package '@vue/compiler-sfc' imported from /layers/heroku_nodejs-pnpm-install/virtual/unplugin-vue-router@0.7.0_vue-router@4.2.5_vue@3.3.7/node_modules/unplugin-vue-router/dist/chunk-QVOU6AJO.mjs

Aside, pnpm install works fine out of the buildpack.

Note that packeto and the old heroku buildpack do not support pnpm, so there isn't a way to switch to another builder.

jose-fully-ported commented 1 year ago

I think one thing the buildpack should do is pass buildpack detection when no packageManager is set but a lockfile is detected but add a warning if so. Ditto if no lockfile but there is a package.json, just show a warning.

colincasey commented 1 year ago

@jose-fully-ported thanks for submitting this issue!

I was able to reproduce your issue with the default Nuxt app + pnpm. This will need to be investigated.

I think one thing the buildpack should do is pass buildpack detection when no packageManager is set but a lockfile is detected but add a warning if so. Ditto if no lockfile but there is a package.json, just show a warning.

We're planning to relax the detection routines for several of the Node.js buildpacks so we can provide warning or errors later in the process.

colincasey commented 11 months ago

@jose-fully-ported Quick update here. I modified the postinstall script to run NODE_DEBUG=module nuxt prepare and got the following output (formatted for readability):

MODULE 311: looking for "@vue/compiler-sfc" in [
  "/layers/heroku_nodejs-pnpm-install/virtual/unplugin-vue-router@0.7.0_vue-router@4.2.5_vue@3.3.8/node_modules/unplugin-vue-router/dist/chunk-QVOU6AJO.mjs/node_modules",
  "/layers/heroku_nodejs-pnpm-install/virtual/unplugin-vue-router@0.7.0_vue-router@4.2.5_vue@3.3.8/node_modules/unplugin-vue-router/dist/node_modules",
  "/layers/heroku_nodejs-pnpm-install/virtual/unplugin-vue-router@0.7.0_vue-router@4.2.5_vue@3.3.8/node_modules/unplugin-vue-router/node_modules",
  "/layers/heroku_nodejs-pnpm-install/virtual/unplugin-vue-router@0.7.0_vue-router@4.2.5_vue@3.3.8/node_modules",
  "/layers/heroku_nodejs-pnpm-install/virtual/node_modules",
  "/layers/heroku_nodejs-pnpm-install/node_modules",
  "/layers/node_modules",
  "/node_modules",
  "/layers/heroku_nodejs-pnpm-install/virtual/nuxt@3.8.1_vite@4.5.0/node_modules/nuxt/bin/node_modules",
  "/layers/heroku_nodejs-pnpm-install/virtual/nuxt@3.8.1_vite@4.5.0/node_modules/nuxt/node_modules",
  "/layers/heroku_nodejs-pnpm-install/virtual/nuxt@3.8.1_vite@4.5.0/node_modules",
  "/layers/heroku_nodejs-pnpm-install/virtual/node_modules",
  "/home/heroku/.node_modules",
  "/home/heroku/.node_libraries",
  "/layers/heroku_nodejs-engine/dist/lib/node"
]

You'll notice that the /workspace/node_modules directory isn't in that list which is why the module load fails even though it is installed.

I've yet to narrow down why the /workspace/node_modules directory is not in the module load path and if that's caused by our pnpm setup, a bug in pnpm or nuxt, or something else.

jose-fully-ported commented 11 months ago

Thanks for following up. Definitely seems weird, does this happen if you run npm or yarn instead đŸ¤” ?

colincasey commented 11 months ago

I'm also able to reproduce the pnpm error outside of the buildpack by copying how the buildpack configures pnpm:

The above steps give the same ERROR Cannot find package '@vue/compiler-sfc' seen in the buildpack.

colincasey commented 11 months ago

@joshwlewis The pnpm docs around the virtual-store-dir configuration appear to be misleading. It gives an example that suggests the virtual-store-dir could be located outside of the project directory but given the comment here, this may not actually work:

Packages only search for modules in node_modules folders that are in parent directories. If you move out the virtual store from your project's directory, where the root modules are symlinked, the packages will not look for it.

I experimented with leaving the virtual-store-dir as the default (node_modules/.pnpm) and this fixes the issue locally but, in the Docker environment, it ends up producing a lot of "cross-device link not permitted" errors.

jose-fully-ported commented 11 months ago

@colincasey this pnpm buildpack for heroku doesn't seem to do much of anything with that property.

Could we just do that instead?

colincasey commented 11 months ago

@jose-fully-ported yes, that pnpm buildpack only uses the store-dir config for caching purposes. The default virtual store (node_modules/.pnpm) would then be recreated on every build. This works fine because these Heroku buildpacks don't execute in a Docker environment so the hard links that pnpm creates between the store and virtual store are allowed.

For CNBs, this is slightly more complicated. You can't create hard links between Docker volumes and the layer cache exists in a different Docker volume. So, if we take that same approach, the cached pnpm store will fail during attempts to hard link and pnpm will fallback to less performant operations when recreating the virtual store. This fixes the issue but produces a lot of output noise which is what I noted in my earlier comment.

jose-fully-ported commented 11 months ago

Maybe related: https://github.com/pnpm/pnpm/issues/6064

Would it be possible to do something like... Create the node_modules dir in a not-volume path and then move it into place for use?

joshwlewis commented 11 months ago

Would it be possible to do something like... Create the node_modules dir in a not-volume path and then move it into place for use?

Yeah, this is pretty similar to the pattern we have in the classic buildpack caching. This would certainly have a performance cost, since we're copying data across mounts.

I've gotten something working locally, where I symlink {virtual_store_layer}/node_modules to /workspace/node_modules, so that the module resolver will be able to find /workspace/node_modules from {virtual_store_layer}/cache/some-dependency by traversing up parent directories.

colincasey commented 11 months ago

Closing. Should be fixed by #737