Next.js API routes (and pages) should support reading files

[Timer]

Feature request

Is your feature request related to a problem? Please describe.

It's currently not possible to read files from API routes or pages.

Describe the solution you'd like

I want to be able to call fs.readFile with a __dirname path and have it "just work".

This should work in Development and Production mode.

Describe alternatives you've considered

This may need to integrate with @zeit/webpack-asset-relocator-loader in some capacity. This plugin handles these types of requires.

However, it's not a necessity. I'd be OK with something that only works with __dirname and __filename (no relative or cwd-based paths).

Additional context

Example:

// pages/api/test.js
import fs from 'fs'
import path from 'path'

export default (req, res) => {
  const fileContent = fs.readFileSync(
    path.join(__dirname, '..', '..', 'package.json'), 
    'utf8'
  )
  // ...
}

Note: I know you can cheat the above example ☝️ with require, but that's not the point. 😄

[robertcoopercode]

@robertcoopercode how did you manage to solve this issue? I am accessing the filesystem using getServerSideProps and process.cwd() which works fine locally but it is still failing on Vercel for me with the following error:

2021-04-28T07:38:22.294Z  060b539a-3bb0-4c61-b009-4870d16fbce4    ERROR   Unhandled error during request: Error: ENOENT: no such file or directory, open '/var/task/content/page.md'
    at Object.openSync (fs.js:476:3)
    at Object.readFileSync (fs.js:377:35)

I finally got it working by moving around where I was calling process.cwd(). Not sure exactly why, but it seems to have a different effect depending on where in your code/functions you call process.cwd(). Not exactly sure what is going on.

[michaelbirchler]

I've successfully used the following code deployed on vercel with next.js 10.1 and webpack 4 inside an API route:

const fileName = path.resolve(
            `./public/`,
            `icons/sdg/${languageShort}/sdg-icon-${req.query.icon}.svg`
)
const svg = fs.readFileSync(fileName)

[ajmeraaxesh]

doesn't seem to be working in Vercel.

[QuentinRoy]

Reading local files from API used to work, but is broken since 10.2.0.

Note that it works in dev mode, but breaks once deployed on Vercel. It seems the build script gets rid of the data files.

[mmazzarolo]

@quentin-sommer any chances you're using Webpack 5 now? I had the same issue when I updated + turned on the Webpack 5 experimental setting.

[quentin-sommer]

I think you found the wrong Quentin 😁

[mmazzarolo]

ouch, sorry! 😅 I meant @QuentinRoy

[QuentinRoy]

@mmazzarolo you are correct. The problem comes from webpack 5.

[eivindml]

Anyone getting this to work? I have disabled webpack 5, and followed instructions here: https://vercel.com/support/articles/how-can-i-use-files-in-serverless-functions But no luck.

[QuentinRoy]

Disabling webpage 5 did the work for me. I did even know about the page you just shared, or did what it suggests.

[SaraVieira]

Can you share your code @QuentinRoy ? I tried and could never get api routes to read the file system

[QuentinRoy]

My code to disable webpack 5?

next.config.js

module.exports = {

  // Rest of the config here...

  future: {
    webpack5: false,
  },
};

You may want to check #24700 for more information.

[alexanderbluhm]

+1 - I also can't get this working

[vithushan19]

+1 also stuck on this. I can access files in the public folder locally, but this breaks when I deploy to vercel.

[BrunoBernardino]

It might help to explain what have you tried. There are a few different things above that seem to have worked for some people.

[sroussey]

OMG, what a waste of several days before finding this. :/

I tried this in my code in a serverless function when deploying nextjs to vercel:

async function* walk(dir) {
    for await (const d of await fs.promises.opendir(dir)) {
      if (d.name === 'node_modules' || d.name === '.git' || d.name === '.next')
        continue;
      const entry = path.join(dir, d.name);
      if (d.isDirectory()) yield* walk(entry);
      else if (d.isFile()) yield entry;
    }
  }

  for await (const p of walk(process.cwd())) console.warn(p);

which shows that not much is there:

2021-05-31T18:19:28.924Z    8fa7fe15-4ef6-47ae-bcea-cb61506471a8    WARN    /var/task/.env
2021-05-31T18:19:28.924Z    8fa7fe15-4ef6-47ae-bcea-cb61506471a8    WARN    /var/task/.env.local.example
2021-05-31T18:19:28.924Z    8fa7fe15-4ef6-47ae-bcea-cb61506471a8    WARN    /var/task/___next_launcher.js
2021-05-31T18:19:28.925Z    8fa7fe15-4ef6-47ae-bcea-cb61506471a8    WARN    /var/task/___vc_bridge.js
2021-05-31T18:19:28.925Z    8fa7fe15-4ef6-47ae-bcea-cb61506471a8    WARN    /var/task/package.json
2021-05-31T18:19:29.023Z    8fa7fe15-4ef6-47ae-bcea-cb61506471a8    ERROR   Error: ENOENT: no such file or 
...

The last line is looking for my data/file.json.

I have tried vercel.json

{
  "functions": {
    "src/pages/**/*": {
      "includeFiles": "data/*"
    }
  }
}

and I have tried next.config.js

const path = require("path");
const CopyPlugin = require("copy-webpack-plugin");

module.exports = {
  future: {
    webpack5: true,
  },
  webpack: (config, { buildId, dev, isServer, defaultLoaders, webpack }) => {
    config.plugins.push(
      new CopyPlugin({
        patterns: ['data/']
      }
    )
   }
}

[sroussey]

Ok, I did a filesystem search and found that the webpack plugin works, but the path is different. Instead of data I can use .next/server/chunks/data

[bapairaew]

For anyone who read the above suggestions and is still confused what to do to make reading static files from inside getServerSideProps and getStaticProps + getStaticPaths with fallback: true works on both Vercel and local.

Here is what I did.

• Put static files in /public e.g. /public/data/item1.json, /public/data/item2.json
• Make a script to copy data files from /public to root directory (e.g. cp -r ./public/data ./data) and execute in on prebuild
• Use path.resolve("./public/data/item1.json") etc to get the data in your code

Now you can use the same code to get data for both your dev mode and production mode. dev mode will use data from ./public. But when you build on Vercel data from root directory will be use. And on Vercel, path.resolve in getServerSideProps will get resolved to wherever ./public locates.

This setup works with webpack 5 and latest Next.js version (at the time of writing it is 10.2.3).

It works for me but the implement does not really feel right so I will look into how can this be better.

You can have a look at this repo for reference: https://github.com/bapairaew/open-fpl/tree/2bfa82b709686330199746859bd3d3c455394261

[tvavrys]

You can do this:

Install copy-webpack-plugin and copy desired files to serverless local filesystem. Then you can locate your files at path.join(process.cwd(), ".next/server/chunks")

next.config.js

const path = require("path")
const CopyPlugin = require("copy-webpack-plugin")

module.exports = {
    target: "serverless",
    future: {
        webpack5: true,
    },
    webpack: function (config, { dev, isServer }) {
        // Fixes npm packages that depend on `fs` module
        if (!isServer) {
            config.resolve.fallback.fs = false
        }
        // copy files you're interested in
        if (!dev) {
            config.plugins.push(
                new CopyPlugin({
                    patterns: [{ from: "content", to: "content" }],
                })
            )
        }

        return config
    },
}

utils.js

import path from "path"
import { promises as fs } from "fs"

export async function getPostBySlug(slug) {
    let basePath = process.cwd()
    if (process.env.NODE_ENV === "production") {
        basePath = path.join(process.cwd(), ".next/server/chunks")
    }

    const filePath = path.join(basePath, `content/${slug}.md`)
    const fileContent = await fs.readFile(filePath, "utf8")
    return fileContent

Thanks @sroussey . You saved me a lot of time.

[leo-petrucci]

I was stuck on this for a while but then I realized that trying to read the filesystem probably isn't the solution i was looking for.

Instead I opted for converting the files I wanted to access (markdown in my case) into JSON at build time and wrote a quick guide for it.

I figure it's more performant than copying dozens of files into the public folder and then parsing them before rendering the API route.

[Janpot]

For relative paths, this works for me under webpack 5:

// pages/api/hello.js
import { readFile } from 'fs/promises';

export default async (req, res) => {
  // reads a file at pages/api/hello.txt
  res.send(await readFile(new URL('./hello.txt', import.meta.url), 'utf-8'));
};

See https://webpack.js.org/guides/asset-modules/#url-assets

edit: This is broken now in v11, see https://github.com/vercel/next.js/pull/27413

[rthaut]

You can do this:

Install copy-webpack-plugin and copy desired files to serverless local filesystem. Then you can locate your files at path.join(process.cwd(), ".next/server/chunks")

...

This was the only solution that worked for me. Thank you @tvavrys for posting it!

I spent a considerable amount of time trying to use the includeFiles functionality in my vercel.json file as described in the documentation, including fighting the correct format for the function pattern (my project would not deploy unless my glob patterns started with "pages/api/", even though all other examples throughout the documentation start with just "api/"). Unfortunately, none of my attempts ever worked when deployed on Vercel.

It would also be nice if there was a way to run Vercel locally, like Netlify offers with Netliify Dev, as that would make troubleshooting issues like these notably easier.

[timonweb]

God bless you @tvavrys, you've just ended my three hour long agony 🙏

[Andrewnt219]

You can do this:

Install copy-webpack-plugin and copy desired files to serverless local filesystem. Then you can locate your files at path.join(process.cwd(), ".next/server/chunks")

next.config.js

const path = require("path")
const CopyPlugin = require("copy-webpack-plugin")

module.exports = {
    target: "serverless",
    future: {
        webpack5: true,
    },
    webpack: function (config, { dev, isServer }) {
        // Fixes npm packages that depend on `fs` module
        if (!isServer) {
            config.resolve.fallback.fs = false
        }
        // copy files you're interested in
        if (!dev) {
            config.plugins.push(
                new CopyPlugin({
                    patterns: [{ from: "content", to: "content" }],
                })
            )
        }

        return config
    },
}

utils.js

import path from "path"
import { promises as fs } from "fs"

export async function getPostBySlug(slug) {
    let basePath = process.cwd()
    if (process.env.NODE_ENV === "production") {
        basePath = path.join(process.cwd(), ".next/server/chunks")
    }

    const filePath = path.join(basePath, `content/${slug}.md`)
    const fileContent = await fs.readFile(filePath, "utf8")
    return fileContent

Thanks @sroussey . You saved me a lot of time.

Need this for my blog. So simple and elegant. Thank you!

[Poujhit]

You can do this:

Install copy-webpack-plugin and copy desired files to serverless local filesystem. Then you can locate your files at path.join(process.cwd(), ".next/server/chunks")

next.config.js

const path = require("path")
const CopyPlugin = require("copy-webpack-plugin")

module.exports = {
    target: "serverless",
    future: {
        webpack5: true,
    },
    webpack: function (config, { dev, isServer }) {
        // Fixes npm packages that depend on `fs` module
        if (!isServer) {
            config.resolve.fallback.fs = false
        }
        // copy files you're interested in
        if (!dev) {
            config.plugins.push(
                new CopyPlugin({
                    patterns: [{ from: "content", to: "content" }],
                })
            )
        }

        return config
    },
}

utils.js

import path from "path"
import { promises as fs } from "fs"

export async function getPostBySlug(slug) {
    let basePath = process.cwd()
    if (process.env.NODE_ENV === "production") {
        basePath = path.join(process.cwd(), ".next/server/chunks")
    }

    const filePath = path.join(basePath, `content/${slug}.md`)
    const fileContent = await fs.readFile(filePath, "utf8")
    return fileContent

Works like a charm!

[okumaru]

You can do this:

Install copy-webpack-plugin and copy desired files to serverless local filesystem. Then you can locate your files at path.join(process.cwd(), ".next/server/chunks")

next.config.js

const path = require("path")
const CopyPlugin = require("copy-webpack-plugin")

module.exports = {
    target: "serverless",
    future: {
        webpack5: true,
    },
    webpack: function (config, { dev, isServer }) {
        // Fixes npm packages that depend on `fs` module
        if (!isServer) {
            config.resolve.fallback.fs = false
        }
        // copy files you're interested in
        if (!dev) {
            config.plugins.push(
                new CopyPlugin({
                    patterns: [{ from: "content", to: "content" }],
                })
            )
        }

        return config
    },
}

utils.js

import path from "path"
import { promises as fs } from "fs"

export async function getPostBySlug(slug) {
    let basePath = process.cwd()
    if (process.env.NODE_ENV === "production") {
        basePath = path.join(process.cwd(), ".next/server/chunks")
    }

    const filePath = path.join(basePath, `content/${slug}.md`)
    const fileContent = await fs.readFile(filePath, "utf8")
    return fileContent

Thanks @sroussey . You saved me a lot of time.

it's been 3 days struggling day and night. and now i very happy my problem have been solved. thank you @tvavrys

[ijjk]

Hi, this is being updated in the latest canary of Next.js and can be tested by enabling experimental: { nftTracing: true } in your next.config.js. Note: process.cwd() should be used instead of __dirname and will point to the current working directory of your Next.js app.

[Svish]

@ijjk Hope there will be some good documentation on how to use this properly too 👍

[kachkaev]

@ijjk that’s great news! I have a couple possibly stupid questions, but it’d be great if you could clarify them for everyone 🙂

1. Does process.cwd() work both in client and server code, just like process.env.NEXT_PUBLIC_XYZ? I.e. is it a webpack replacement trick meaning that it stops working if I do const x = process; console.log(x.cwd())?

2. Do I need to worry about the actual working directory or will the value be ‘stable’ regardless of what real cwd is? Here is a somewhat artificial example to illustrate what I mean:

cd /path/to/project
cd node_modules/.bin
./next 
## Is process.cwd() in Next.js pages and API routes
##   /path/to/project
## or 
##   /path/to/project/node_modules/.bin
## ?

[rodrigoKulb]

Hi, this is being updated in the latest canary of Next.js and can be tested by enabling experimental: { nftTracing: true } in your next.config.js. Note: process.cwd() should be used instead of __dirname and will point to the current working directory of your Next.js app.

for me it worked

[ijjk]

@kachkaev this is specific to reading files inside server-side methods like API routes or getStaticProps/getServerSideProps. For client-side file loading new URL() should be used with webpack 5.

The process.cwd() value isn't modified during the build and will be set to where next start is run which should be the Next.js project's base directory.

[elrumordelaluz]

In case could be useful for someone to try, for me worked a combination of some comments, after lots of deploys.

// next.config.js
module.exports = {
  target: 'serverless',
  experimental: { nftTracing: true },
}

The folder with files called by the function inside /public, and invoked like:

resolve('./public/my-folder/my-file.txt')

[pgrodrigues]

After some tries, this works for me on next 11.1.3-canary.32.

Accessing template.mjml in email/ dir:

// next.config.js
module.exports = {
   experimental: { outputFileTracing: true }
}

// vercel.json
{
   "functions": {
      "pages/api/contact.js": {
         "includeFiles": "email/template.mjml"
      }
   }
}

// pages/api/contact.js
const { join, resolve } = require("path");
const { readFileSync } = require("fs");

export default async (req, res) => {
   const templateDirectory = resolve(process.cwd(), "email");
   const emailTemplate = readFileSync(join(templateDirectory, "template.mjml"), "utf8");

  // ...
};

[lazlothemonkey]

You can do this:

Install copy-webpack-plugin and copy desired files to serverless local filesystem. Then you can locate your files at path.join(process.cwd(), ".next/server/chunks")

next.config.js

const path = require("path")
const CopyPlugin = require("copy-webpack-plugin")

module.exports = {
    target: "serverless",
    future: {
        webpack5: true,
    },
    webpack: function (config, { dev, isServer }) {
        // Fixes npm packages that depend on `fs` module
        if (!isServer) {
            config.resolve.fallback.fs = false
        }
        // copy files you're interested in
        if (!dev) {
            config.plugins.push(
                new CopyPlugin({
                    patterns: [{ from: "content", to: "content" }],
                })
            )
        }

        return config
    },
}

utils.js

import path from "path"
import { promises as fs } from "fs"

export async function getPostBySlug(slug) {
    let basePath = process.cwd()
    if (process.env.NODE_ENV === "production") {
        basePath = path.join(process.cwd(), ".next/server/chunks")
    }

    const filePath = path.join(basePath, `content/${slug}.md`)
    const fileContent = await fs.readFile(filePath, "utf8")
    return fileContent

Thanks @sroussey . You saved me a lot of time.

Hello @tvavrys ,

I tried this approach. It works when I use npm run build locally but I when I deploy to vercel I keep getting this error:

2021-09-25T14:19:26.441Z    41890285-cb02-4c0d-9521-41cd89e9e6b6    ERROR   [Error: ENOENT: no such file or directory, open '/var/task/.next/serverless/chunks/textbooks/boledu-hls-textbook-main/markdown/1-introduction/README.md'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/var/task/.next/serverless/chunks/textbooks/boledu-hls-textbook-main/markdown/1-introduction/README.md'
}

I did change the base path to this:

if (process.env.NODE_ENV === "production") {
  basePath = path.join(process.cwd(), ".next/serverless/chunks")
}

so server to serverless (otherwise it doesn't work on npm run build)

and the next.config:

const path = require("path");
const CopyPlugin = require("copy-webpack-plugin");

module.exports = {
  target: "serverless",

  webpack5: true,

  webpack: function (config, { dev, isServer }) {
    // Fixes npm packages that depend on `fs` module
    if (!isServer) {
      config.resolve.fallback.fs = false;
    }
    // copy files you're interested in
    if (!dev) {
      config.plugins.push(
        new CopyPlugin({
          patterns: [{ from: "textbooks", to: "textbooks" }],
        })
      );
    }

    return config;
  },
};

Would you have any idea what could be breaking it? Thanks.

[masstapel]

@ijjk I'm using something similar in our project right now: During build time, we receive data from a CMS which we write in a JSON file to be read during build and run time. The directory structure is ./json// (so located in root of project).

Everything seems to work during build and runtime both locally and deployed, except for revalidation: For some reason, process.cwd() returns a different value when revalidation of a ISR page is happening.

During revalidation when a new file needs to be generated, we first use fs.unlink to remove previous files and then we create and write the new one. In this step, process.cwd() returns /var/task/json as a path while during build time this returns /vercel/path0/json which is the actual location of the cached files when they got created during build time. Therefore, the content never gets revalidated correctly and revalidation fails.

Do you have a good suggestion for this use case? Many thanks!

[revskill10]

So, the question is, how to access current directory of a page ? For example, i'm in the page /pages/[postId]/index.js, how to get this with process.cwd() ?

[sroussey]

That's a different issue, and really, the directory does not exist as the JS. Ode gets bundled into one file.

[Richard87]

Hi, I have the same error, with @grpc/client unable to load *.proto* files when target is serverless.

I created a basic repo to reproduce the error: https://github.com/richard87/test-grpc Basically, running in dev "works" (get a error that it cant connect to the server), but running build && start, returning a autopilot.proto file not found error... check the /api/hello to test it out.

[bmadsen-ownrs]

I'm on Next.js 12 and I previously used @tvavrys's suggestion but this seems to be broken now. I have a serverless function that reads script files relative to the project's root and runs them when invoked from a Next.js page via http calls.

It seems that exporting a unstable_includeFiles config prop for the serverless api file works:

// in /pages/api/some-route.ts
export const config = {
  unstable_includeFiles: ['folder-relative-to-root'],
};

and then a path to the file is simply

const basePath = process.cwd();
const scriptsDir = path.join(basePath, 'folder-relative-to-root');

🤷 works but seems to be a temporary hack ?

[curly210102]

https://github.com/vercel/next.js/issues/8251#issuecomment-927114789

It works with outputFileTracing in nextjs v12.0.2

// next.config.js
module.exports = {
   outputFileTracing: true
}

But, it doesn't work when file is "README.md", it's really weird. @ijjk

Code as below:

// vercel.json
{
   "functions": {
      "pages/api/contact.js": {
         "includeFiles": "extensions/**/README.md"
      }
   }
}

// pages/api/contact.js
import type { NextApiRequest, NextApiResponse } from "next";
import { resolve, join } from "path";
import { readFileSync } from "fs";

type Data = {
  content: string;
};

type Error = {
  error: string;
};

export default function handler(
  req: NextApiRequest,
  res: NextApiResponse<Data | Error>
) {
  if (req.method === "GET") {
    const { slugs } = req.query;
    const slug = typeof slugs === "string" ? slugs : slugs[0];

    const templateDirectory = resolve(process.cwd(), "extensions");
    const emailTemplate = readFileSync(
      join(templateDirectory, slug, "README.md"),
      "utf8"
    );
    return res.status(200).json({
      content: emailTemplate,
    });
  } else {
    return res.status(404).json({
      error: "Not Support Method",
    });
  }
}

• Reproduction Demo Repo: https://github.com/curly210102/next-js-issue
• Test URL: https://next-js-issue.vercel.app/api/contact?slugs=a
• Vercel Function Log:

[ijjk]

@curly210102 includeFiles in vercel.json isn't used by Next.js. It looks like the readFile call in the above snippet is using the wrong path it's reading from extensions/README.md but the readme is actually located at extensions/a in the repro https://github.com/curly210102/next-js-issue/tree/main/extensions/a

[curly210102]

@curly210102 includeFiles in vercel.json isn't used by Next.js. It looks like the readFile call in the above snippet is using the wrong path it's reading from extensions/README.md but the readme is actually located at extensions/a in the repro https://github.com/curly210102/next-js-issue/tree/main/extensions/a

@ijjk Oh sorry, the code snippet pasted in the above with a accidental omission, please take the repo as the prevailing.

To illustrate the specific behavior of "README.md", I added "index.md" demo as a comparison. And remove the vercel.json.

• Reproduction Demo Repo: https://github.com/curly210102/next-js-issue
• README.md Test URL: https://next-js-issue.vercel.app/api/contact?slugs=a
• index.md Test URL: https://next-js-issue.vercel.app/api/info?slugs=a

[QB3L]

Is there a solution for this? We need to be able to read some template files as well in an endpoint under pages/api

@curly210102 did you find anything that could resolve the issue in your example?

[mtimofiiv]

I have a slightly different permutation of the same problem. I reproduced it here:

https://github.com/mtimofiiv/vercel-fonts-demo

And it's deployed here: https://vercel-fonts-demo.vercel.app

Readme file in the repo has all the things I tried – but basically I followed a bunch of different instructions from this thread plus also this one:

https://github.com/lovell/sharp/issues/2499

[lonniev]

When the fs.readFileSync ENOENT` (file not found) issue occurs, it may happen for primarily two reasons here:

1. The path to the sought file is incorrect
2. The file truly is absent at the path as provided

For (1), this may be due to relocation of the files in the process of deployment from development locations to operations locations.

For (2), this can be due to pilot error (of course) or to incomplete Next.js configuration.

When I encountered this issue, I was trying to read from the file system a GraphQL schema file. Those files are lightly-stylized blends of JSON and Javascript/Typescript - but they use a .graphql suffix.

My local development build and run was fine but the schema.graphql file that is imported during configuration of the Apollo GraphQL Server was reported as ENOENT when the project was deployed to Vercel.

Apparently, the *.graphql file was not getting deployed to the server-less location because it uses an atypical suffix.

See https://nextjs.org/docs/api-reference/next.config.js/custom-page-extensions

Here’s the crucial sample advice:

Open next.config.js and add the pageExtensions config:

module.exports = {
  pageExtensions: ['mdx', 'md', 'jsx', 'js', 'tsx', 'ts'],
}

Note: The default value of pageExtensions is ['tsx', 'ts', 'jsx', 'js']. 

Also be advised that when we modify the set of file extensions to be considered, we have to restate the default extensions - or else the files which were deployed will then get excluded - leaving a broken mess.

(now to test this in my repo. If this is all true, I can remove some workaround stuff that might have worked but seems to have broken Dynamic Routing.)

Update: this may help some people - but it isn't the right change for me. If I add qraphql as a page extension, then next.js attempts to parse the file, on its own, as legitimate code.

Update: I gave up on attempting to get Next.js and Vercel to be able to deploy the non-code file to the pages/api area and to run the server less functions with next. For now, I simply store the content of the graphql.schema file as a string in the Javascript file that was trying to load the file.

This was an educational diversion into the details of Next.js and Vercel but isn't the major goal of what I am prototyping.

I'll monitor for updates to this issue and refactor the bandaid area later.

[mdstroebel]

So I have had this same issue and I recently realised that I can just use Webpack Copy Plugin to enforce copying the required files to the bundle (within the .next folder for NextJS).

However, on deployment to Vercel, it is not adding/hosting those files when I check the Source -> Output UI in Vercel. See screenshot of that UI without my custom templates folder (which is bundled correctly within the .next folder locally when I run npm run build).