With webpack 5 and once deployed, local files are not readable from APIs anymore

QuentinRoy commented 3 years ago

What version of Next.js are you using?

10.2.0

What version of Node.js are you using?

14.0.0

What browser are you using?

Chrome,Firefox,Safari

What operating system are you using?

macOS

How are you deploying your application?

Vercel

Describe the Bug

Since updating to 10.2.0, local files are not accessible from apis anymore once deployed on Vercel. I do not know if it was on purpose, but it used to work fine before upgrading.

In both case, an internal server error is returned. After closer inspection of the logs, the error message is as follows:

[Error: ENOENT: no such file or directory, open '/app/data/metadata.yml'] {

errno: -2,

code: 'ENOENT',

syscall: 'open',

path: '/app/data/metadata.yml'

}

It went back to normal after downgrading to next 10.1.3.

EDIT: It seems the problem comes from webpack 5, it is working fine if webpack5 is disabled in next.config.js.

EDIT: It does not fail if there is only one API trying to read only one local file. It is possible the problem occurs when multiple files have to be available for reading.

EDIT: According to @jacksonhardaker, this also fails when reading files from getServerSideProps.

Expected Behavior

Local files should be readable from apis. If these are not embedded by default anymore, maybe an option could be used to indicate what files should be available?

To Reproduce

Here is an example showing APIs failing: https://github.com/QuentinRoy/vercel-local-min-example deployed at https://vercel-local-min-example.vercel.app.

Here is another example from @jacksonhardaker showing getServerSideProps failing: https://github.com/jacksonhardaker/next-webpack-5-fs-bug deployed at https://next-webpack-5-fs-bug-jwegk8q1o-jacksonhardaker1.vercel.app/.

QuentinRoy commented 3 years ago

Following @mmazzarolo suggestion in #8251, I can attest the problem actually comes with webpack 5. Disabling webpack 5 after updating fixes it:

next.config.js

module.exports = {
  future: {
    webpack5: false,
  },
};

EDIT: With newer Next versions, switching back to webpack 4 is:

module.exports = {
  webpack5: false,
};

joe-bell commented 3 years ago

I manually opted into webpack5 and can confirm I'm also seeing this behaviour inside API routes

Example

API Route

const CONTENT_DIR = path.join(process.cwd(), "src", "content");

export const getContentByDir = async (
  dir: string,
  options?: IGetContentOptions
) => {
  const srcDir = path.join(CONTENT_DIR, dir);

  if (!fs.lstatSync(srcDir)?.isDirectory()) {
    // Error thrown to Vercel
    throw Error(`${srcDir} is not a valid directory`);
  }
  // other unrelated code
};

Vercel Logs

Error: ENOENT: no such file or directory, lstat '/var/task/src/content/blog'
    at Object.lstatSync (fs.js:1077:3)
    at getContentByDir (/var/task/.next/server/pages/api/sitemap.js:283:49)
    at getBlogPaths (/var/task/.next/server/pages/api/sitemap.js:292:23)
    at getAllPaths (/var/task/.next/server/pages/api/sitemap.js:313:67)
    at getSitemapPaths (/var/task/.next/server/pages/api/sitemap.js:314:43)
    at exports.modules.3692.__webpack_exports__.default (/var/task/.next/server/pages/api/sitemap.js:331:96)
    at apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:8:7)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Server.handleApiRequest (/var/task/node_modules/next/dist/next-server/server/next-server.js:66:462)
    at async Object.fn (/var/task/node_modules/next/dist/next-server/server/next-server.js:58:580) {
  errno: -2,
  syscall: 'lstat',
  code: 'ENOENT',
  path: '/var/task/src/content/blog'
}

Additional Context

I tried adding this to my next.config.js, but had no luck:

  webpack: (config, { isServer }) => {
    if (!isServer) {
      config.resolve.fallback.fs = false;
    }

    return config;
  },

This behaviour doesn't occur when I opt-out of webpack5

ijjk commented 3 years ago

Hi, can someone who's run into this provide a repo with a minimal reproduction, we aren't able to reproduce this with the above provided information

QuentinRoy commented 3 years ago

Hi @ijjk, thank you for looking into this. I investigated the problem a bit more. It seems it is working fine when there is only one api reading one local file. However, if there are two apis reading two local files, it breaks. Here is a minimal reproduction repository: https://github.com/QuentinRoy/vercel-local-min-example. This is deployed here: https://vercel-local-min-example.vercel.app

jacksonhardaker commented 3 years ago

Good tip @QuentinRoy I can confirm that the addition of a second read appears to reveal the issue. Repo: https://github.com/jacksonhardaker/next-webpack-5-fs-bug Deployment: ~https://next-webpack-5-fs-bug-mfyukza5m-jacksonhardaker1.vercel.app/~ https://next-webpack-5-fs-bug-jwegk8q1o-jacksonhardaker1.vercel.app/

edit: updated the deployment url

wworrall commented 3 years ago

This is a major issue causing Vercel deployments to break silently (build completing successfully). Rolling back to webpack 4 for now - please fix asap, Vercel 😍. Also, perhaps add something to your tests as to which projects are automatically compatible with webpack5 https://nextjs.org/docs/messages/webpack5

joe-bell commented 3 years ago

@wworrall your tone is super unproductive and contributes nothing to this issue

QuentinRoy commented 3 years ago

@jacksonhardaker, does your deployment matches the code in your repo? The deployed app appears functional.

jacksonhardaker commented 3 years ago

@QuentinRoy Thanks for the ping, I must have pasted the incorrect url. Updated.

marbiano commented 3 years ago

We have a slightly similar case. We can read files but only if one API endpoint is doing the fs.readFile. The moment we add a second endpoint it fails!

Thanks for taking a look and providing help, Vercel crew. 🙏

EDIT: just realized it's the same issue as @QuentinRoy described above. Good to know we are not the only ones!

nath-green commented 3 years ago

I'm not sure if this is related but the value of process.cwd() changes after first render to a refresh. From /vercel/path0 to /var/task which causes my files not to be read after a refresh as the path is different.

One thing I did notice is that if the revalidate prop is removed, it appears to work correctly.

ijjk commented 3 years ago

Hi, this has been updated and can now be tested in v11.1.1-canary.14 of Next.js by adding the below config to your next.config.js and re-deploying. Please update and give it a try!

module.exports = {
  experimental: {
    nftTracing: true
  }
}

joe-bell commented 3 years ago

@ijjk just tried testing this locally and deployed to Vercel – I can confirm everything works as expected now. Thank you!

marbiano commented 3 years ago

Yeah, I can confirm that it's working like charm!

jonalvarezz commented 3 years ago

Thanks for the update @ijjk

Though it didn't work for me on production on Vercel.

Not sure what I am missing. This page will fail to load and here is the code (branch prod-i18n; running on v11.1.1-canary-14; nftTracing enabled.)

I've tried everything proposed here with no luck. I had to disable i18n to get it to work on Vercel.

I also tried next.js v11.1.2

carl0s commented 3 years ago

Any update on this?

ijjk commented 3 years ago

@carl0s did you try the experimental.nftTracing config already and experienced issues still? If so you can try the latest version of Next.js v11.1.3-canary.45 with the now renamed experimental.outputFileTracing.

carl0s commented 3 years ago

I will totally try that!

carl0s commented 3 years ago

I tried that with the mentioned version of Next.js, v11.1.3-canary.45. I'm trying to have a next build with next-18n and I have this error.

Error occurred prerendering page "/de/article/test-article". Read more: https://nextjs.org/docs/messages/prerender-error Error: Cannot find module '[redacted]/next-i18next.config.js' at webpackEmptyContext ([redacted]/.next/serverless/pages/[...slug].js:11:10) at [redacted]/.next/serverless/chunks/709.js:245:72 at processTicksAndRejections (node:internal/process/task_queues:96:5)

I removed some reference to local paths, as they might be confusing. Without I18n works fine, and with webpack 4 it builds fine. Though we need webpack 5 to work correctly.

adrai commented 3 years ago

I tried that with the mentioned version of Next.js, v11.1.3-canary.45. I'm trying to have a next build with next-18n and I have this error.

Error occurred prerendering page "/de/article/test-article". Read more: https://nextjs.org/docs/messages/prerender-error Error: Cannot find module '[redacted]/next-i18next.config.js' at webpackEmptyContext ([redacted]/.next/serverless/pages/[...slug].js:11:10) at [redacted]/.next/serverless/chunks/709.js:245:72 at processTicksAndRejections (node:internal/process/task_queues:96:5)

I removed some reference to local paths, as they might be confusing. Without I18n works fine, and with webpack 4 it builds fine. Though we need webpack 5 to work correctly.

@carl0s This is a "known" issue... read about it here: https://github.com/isaachinman/next-i18next/issues/1202#issuecomment-869209088 and here: https://github.com/isaachinman/next-i18next/pull/1384

eric-burel commented 3 years ago

The update to 11.3 with experimental.outputFileTracing seems to stop this error message from happening but leads to new issues with Material UI v5 / Emotion, I get this:

/node/index.js',
    '/var/task/node_modules/@mui/system/index.js',
    '/var/task/node_modules/@mui/material/node/styles/adaptV4Theme.js',
    '/var/task/node_modules/@mui/material/node/styles/index.js',
    '/var/task/.next/server/pages/_document.js',
    '/var/task/node_modules/next/dist/server/require.js',
    '/var/task/node_modules/next/dist/server/load-components.js',
    '/var/task/node_modules/next/dist/server/api-utils.js',
    '/var/task/node_modules/next/dist/server/next-server.js',
    '/var/task/___next_launcher.js',
    '/var/runtime/UserFunction.js',
    '/var/runtime/index.js'
  ]
}
2021-10-15T07:57:31.238Z    f5ecb81a-7cb9-49fa-9f66-4595e6f427d0    ERROR   Error: Cannot find module 'stylis'
Require stack:
- /var/task/node_modules/@emotion/cache/dist/emotion-cache.cjs.prod.js
- /var/task/node_modules/@emotion/cache/dist/emotion-cache.cjs.js
... // etc.

Seems similar to this ticket: https://github.com/emotion-js/emotion/issues/850 and this old SO question (https://stackoverflow.com/questions/62526318/nextjs-emotionjs-and-vercel-error-err-require-esm-must-use-import-to-load-es). Installing stylis explicitely doesn't fix the issue.

Only one route of my app is affected I don't really get why specifically, probably because it's an MDX file, with @next/mdx. It also affects API routes despite them not using @emotion at all.

I can reproduce only on Vercel so far, not locally. I can share a non-minimal repro if needed, this affect an open source project.

ijjk commented 3 years ago

@eric-burel the experimental.outputFileTracing has been updated in the latest version of Next.js v11.1.3-canary.86, please update and give it a try! If you are still seeing issues please provide the mentioned repo for further investigation.

adrai commented 3 years ago

v11.1.3-canary.86 behaves still the same for next-i18next issue: https://github.com/vercel/next.js/issues/24700#issuecomment-935735270

eric-burel commented 3 years ago

@ijjk First try after this update canary.86 gives me ModuleNotFoundError: Module not found: Error: Can't resolve 'querystring' in '/vercel/path0/node_modules/next/dist/compiled/native-url' during Vercel build and locally as well. It's probably just an unrelated regression in this version, it's fixed by installing querystring, explicitely.

Then after a second try, it seems to partially fix the issue: I don't have trouble with API routes anymore.

But my MDX page still doesn't work.

2021-10-20T15:24:05.039Z    2f2bc57a-9453-45f1-8b7c-785d883c2e14    ERROR   TypeError: Cannot destructure property 'components' of 'object null' as it is null.
    at Server.renderToResponseWithComponents (/var/task/node_modules/next/dist/server/next-server.js:985:90)
    at Server.renderErrorToResponse (/var/task/node_modules/next/dist/server/next-server.js:1428:35)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Server.pipe (/var/task/node_modules/next/dist/server/next-server.js:866:25)
    at async Object.fn (/var/task/node_modules/next/dist/server/next-server.js:671:21)
    at async Router.execute (/var/task/node_modules/next/dist/server/router.js:205:32)
    at async Server.run (/var/task/node_modules/next/dist/server/next-server.js:843:29)
    at async Server.handleRequest (/var/task/node_modules/next/dist/server/next-server.js:293:20)
    at async Server.<anonymous> (/var/task/___next_launcher.js:32:9)

This page works locally but still not in Vercel.

You can get the code here: https://github.com/eric-burel/vulcan-next/tree/main Faulty page is there: https://vulcan-next.vercel.app/learn

ijjk commented 3 years ago

@adrai it looks like the config is being included correctly in the latest canary while deploying the example, test deployment done here it seems there is a different issue which is responded about on the initial issue here.

ijjk commented 3 years ago

@eric-burel it looks like your issue is related to next-i18next as well so the above comment is relevant for your case as well.

adrai commented 3 years ago

@adrai it looks like the config is being included correctly in the latest canary while deploying the example, test deployment done here it seems there is a different issue which is responded about on the initial issue here.

@ijjk I still get: error - Error: Cannot find module '/Users/usr/Projects/next-i18next/examples/simple/next-i18next.config.js'

steps to reproduce:

clone https://github.com/isaachinman/next-i18next
add experimental flag here: https://github.com/isaachinman/next-i18next/blob/master/examples/simple/next.config.js
use new canary version of next here: https://github.com/isaachinman/next-i18next/blob/master/examples/simple/package.json#L12
npm run run-example and visit http://localhost:3000

ijjk commented 3 years ago

@adrai that appears to be an issue with running the package from the monorepo, the package.json for the example should look like this:

{
  "name": "v10",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "scripts": {
    "dev": "next",
    "build": "next build",
    "start": "next start"
  },
  "dependencies": {
    "next": "canary",
    "next-i18next": "latest",
    "react": "^17.0.2",
    "react-dom": "^17.0.2"
  },
  "devDependencies": {
    "prop-types": "^15.7.2"
  }
}

and the modules should be installed specific to the example along with running yarn next build inside of the example folder. The reason it isn't finding the config is because an incorrect current working directory is being used.

adrai commented 3 years ago

@ijjk you're right

eric-burel commented 3 years ago

Much thanks for the help @ijjk I've made some progress.

I ended up with the config described there in the relevant next-i18next ticket https://github.com/isaachinman/next-i18next/issues/661#issuecomment-947945079

However, I have a new issue :/

2021-10-20T18:55:03.846Z    63b14614-4efe-45fb-877f-d362013ab373    ERROR   Error: Cannot find module '@mdx-js/react'
Require stack:
- /var/task/.next/server/pages/learn/intro-online.js
- /var/task/node_modules/next/dist/server/require.js
- /var/task/node_modules/next/dist/server/load-components.js
- /var/task/node_modules/next/dist/server/api-utils.js
- /var/task/node_modules/next/dist/server/next-server.js
- /var/task/___next_launcher.js
- /var/runtime/UserFunction.js
- /var/runtime/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
    at Function.Module._load (internal/modules/cjs/loader.js:745:27)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at Object.7425 (/var/task/.next/server/pages/learn/intro-online.js:144:18)
    at __webpack_require__ (/var/task/.next/server/webpack-runtime.js:25:42)
    at Object.8373 (/var/task/.next/server/pages/learn/intro-online.js:19:71)
    at __webpack_require__ (/var/task/.next/server/webpack-runtime.js:25:42)
    at __webpack_exec__ (/var/task/.next/server/pages/learn/intro-online.js:399:39)
    at /var/task/.next/server/pages/learn/intro-online.js:400:91 {
  code: 'MODULE_NOT_FOUND',

this happens if I refresh the page https://vulcan-next.vercel.app/learn/intro-online
this doesn't happen if I go to another page, and just add /learn in the URL => as far as I can tell Next is doing a kind of client page transition in this case. So basically only the server render fails.
no repro locally, it works ok

The package is correctly listed in the dependencies.

Sorry I'd like to help more but I don't fully understand what happens, and the error message is kinda vague here. That's maybe a different issue. Could there be some server-rendering happening? This is an MDX page using the row @next/mdx setup and MDXProvider from @mdx-js/react to pass some components.

ijjk commented 3 years ago

@eric-burel you are adding getInitialProps in _app here so any page that doesn't use getStaticProps will be server rendered for every request. It seems the above shared URL is working correctly currently, did you need to change something related to mdx for this to work?

ijjk commented 3 years ago

For anyone else who ran into the initial issue, please try upgrading to v11.1.3-canary.104 of Next.js and remove the experimental.outputFileTracing as it is no longer needed and confirm the issue is resolved fully 🙏

eric-burel commented 3 years ago

@ijjk Nope it doesn't actually work, if you reload the page again, you'll get a 500 error. After updating to canary 104 I still get the server error Cannot find module '@mdx-js/react'. Very weird, I could not reproduce locally ever, it's really specific to the Vercel deployment.

Thanks for the tip though, I think getting rid of getInitialProps will fix it since it will prevent a rerender, I need to update next-i18next to the latest version. However it also means I'll lose next export, but I can live without it. I'll do that and open a separate issue if I'm still stuck.

ijjk commented 2 years ago

I'm going to close this as this should be working properly in the latest version of Next.js v12.0.4, if you are still encountering issues with this in the latest version please open a fresh issue with reproduction.

balazsorban44 commented 2 years ago

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.

vercel / next.js