search

nrwl / nx

Smart Monorepos · Fast CI
https://nx.dev
MIT License
23.78k stars 2.37k forks source link

Next.js standalone output broken when cached due to missing symlinks #19502

Closed ianldgs closed 8 months ago

ianldgs commented 1 year ago

Current Behavior

Symlinks are not restored when cache is used:

❯ ls -lah dist/apps/my-project/.next/standalone/node_modules/

total 0
drwxr-xr-x   11 iserpa  staff   352B Oct  9 14:33 .
drwxr-xr-x    6 iserpa  staff   192B Oct  9 14:33 ..
drwxr-xr-x  117 iserpa  staff   3.7K Oct  9 14:33 .pnpm
drwxr-xr-x    4 iserpa  staff   128B Oct  9 14:33 babel-plugin-macros
drwxr-xr-x    5 iserpa  staff   160B Oct  9 14:33 caniuse-lite
drwxr-xr-x    7 iserpa  staff   224B Oct  9 14:33 next
drwxr-xr-x    5 iserpa  staff   160B Oct  9 14:33 react
drwxr-xr-x    6 iserpa  staff   192B Oct  9 14:33 react-dom
drwxr-xr-x    5 iserpa  staff   160B Oct  9 14:33 sass
drwxr-xr-x    5 iserpa  staff   160B Oct  9 14:33 uglify-js
drwxr-xr-x    6 iserpa  staff   192B Oct  9 14:33 webpack

Expected Behavior

When cache is used, symlinks should also be restored:

❯ ls -lah dist/apps/my-project/.next/standalone/node_modules/

total 0
drwxr-xr-x   11 iserpa  staff   352B Oct  9 14:31 .
drwxr-xr-x    6 iserpa  staff   192B Oct  9 14:31 ..
drwxr-xr-x  117 iserpa  staff   3.7K Oct  9 14:31 .pnpm
lrwxr-xr-x    1 iserpa  staff    64B Oct  9 14:31 babel-plugin-macros -> .pnpm/babel-plugin-macros@3.1.0/node_modules/babel-plugin-macros
lrwxr-xr-x    1 iserpa  staff    57B Oct  9 14:31 caniuse-lite -> .pnpm/caniuse-lite@1.0.30001492/node_modules/caniuse-lite
lrwxr-xr-x    1 iserpa  staff    63B Oct  9 14:31 next -> .pnpm/next@13.4.19_pamseawcqpaoyqfggclsvtomea/node_modules/next
lrwxr-xr-x    1 iserpa  staff    37B Oct  9 14:31 react -> .pnpm/react@18.2.0/node_modules/react
lrwxr-xr-x    1 iserpa  staff    58B Oct  9 14:31 react-dom -> .pnpm/react-dom@18.2.0_react@18.2.0/node_modules/react-dom
lrwxr-xr-x    1 iserpa  staff    35B Oct  9 14:31 sass -> .pnpm/sass@1.60.0/node_modules/sass
lrwxr-xr-x    1 iserpa  staff    45B Oct  9 14:31 uglify-js -> .pnpm/uglify-js@3.17.4/node_modules/uglify-js
lrwxr-xr-x    1 iserpa  staff    68B Oct  9 14:31 webpack -> .pnpm/webpack@5.88.2_l5cbnh2nwjrm2cqen44lw7rug4/node_modules/webpack

GitHub Repo

No response

Steps to Reproduce

  1. nx build my-project --skip-nx-cache
  2. rm -rf dist/apps/my-project/
  3. nx build my-project
  4. nx docker my-project (https://github.com/gperdomor/nx-tools/tree/main/packages/nx-container)
  5. docker run my-registry/my-project

Nx Report

>  NX   Report complete - copy this into the issue template

   Node   : 18.16.1
   OS     : darwin-arm64
   pnpm   : 7.24.3

   nx                 : 16.10.0
   @nx/js             : 16.10.0
   @nx/jest           : 16.10.0
   @nx/linter         : 16.10.0
   @nx/workspace      : 16.10.0
   @nx/cypress        : 16.10.0
   @nx/devkit         : 16.10.0
   @nx/eslint-plugin  : 16.10.0
   @nx/next           : 16.10.0
   @nx/plugin         : 16.10.0
   @nx/react          : 16.10.0
   @nx/rollup         : 16.10.0
   @nx/storybook      : 16.10.0
   @nrwl/tao          : 16.10.0
   @nx/web            : 16.10.0
   @nx/webpack        : 16.10.0
   typescript         : 5.1.6
   ---------------------------------------
   Community plugins:
   @jscutlery/semver           : 3.1.0
   @nx-aws-plugin/nx-aws-cache : 3.1.0
   @nx-tools/nx-container      : 5.0.3
   @nxkit/playwright           : 3.0.2
   ngx-deploy-npm              : 7.0.1
   ---------------------------------------
   Local workspace plugins:
         @imc-trading/webgui
         @imc-trading/nx-plotly
         @imc-trading/nx-web

Failure Logs

This causes problems with module resolution when running the production docker build of the application:

Package Manager Version

Tried with pnpm@7.24.3 and pnpm@8.8.0

Operating System

Additional Information

The offending change seems to be https://github.com/nrwl/nx/pull/17638.

copy doesn't use node:fs-extra anymore: https://github.com/nrwl/nx/pull/17638/files#diff-487f8cf4d7fba58c204044696a570374a67661d9489b2e9a6182689e411ecb23R168

it calls copy_items from rust fs_extra crate: https://github.com/nrwl/nx/pull/17638/files#diff-663ac081b66b48c93df5278df020d105321ea28dc22e143efcbad9888727d153R22.

which doesn't seem to support symlinks: https://github.com/webdesus/fs_extra/issues/15

ianldgs commented 1 year ago

@AgentEnder / @FrozenPandaz any comment on this? Is anyone gonna work on a fix soon? Maybe for v17? If not, can you please suggest a fix that I can implement? This is blocking us from upgrading. Stuck in 16.3; next working version would be 16.4, but we had other issues in that version that were fixed later on.

ianldgs commented 1 year ago

@Cammisuli maybe you have some insight, as the author of https://github.com/nrwl/nx/pull/17638?

Cammisuli commented 1 year ago

This is an interesting scenario.. Let me discuss this with a few others. Just for my knowledge, what would you do when things are restored from the remote cache / Nx Cloud?

ianldgs commented 1 year ago

This is an interesting scenario.. Let me discuss this with a few others. Just for my knowledge, what would you do when things are restored from the remote cache / Nx Cloud?

We build a docker image by copying the output on dist/apps/my-project.

The advantage here is that we don't need to run pnpm install in the Dockerfile of each app, since all dependencies are already downloaded and bundled.

But good to know about this limitation. Not sure what else can be broken without symlink support.

ToppScorer commented 11 months ago

We have the same problem (described here https://stackoverflow.com/questions/77627408/does-nx-support-caching-symbolic-links) and also think that this should be supported again (I found this issue after creating the question in stackoverflow).

Chanki-Min commented 10 months ago

This is acts as quite blocker to next.js developers. nx cache should be able to restore same build artifacts. I know nx team has busy days, but is there any plan to solve this problem? Seems like this issue is staled @Cammisuli

jchamale commented 8 months ago

Any updates? We're blocked currently because our current Next standalone is copying the files and not the symlinks

doong-jo commented 8 months ago

@AgentEnder @FrozenPandaz any comment on this?

p-iknow commented 8 months ago

@FrozenPandaz

Wow, this bug has finally been fixed. I was having a hard time because of this issue, but I'm so grateful that it's been resolved.

Chanki-Min commented 8 months ago

Omg it FIXED!!! thanks @FrozenPandaz

github-actions[bot] commented 6 months ago

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.