Next.js API routes (and pages) should support reading files

[Timer]

Feature request

Is your feature request related to a problem? Please describe.

It's currently not possible to read files from API routes or pages.

Describe the solution you'd like

I want to be able to call fs.readFile with a __dirname path and have it "just work".

This should work in Development and Production mode.

Describe alternatives you've considered

This may need to integrate with @zeit/webpack-asset-relocator-loader in some capacity. This plugin handles these types of requires.

However, it's not a necessity. I'd be OK with something that only works with __dirname and __filename (no relative or cwd-based paths).

Additional context

Example:

// pages/api/test.js
import fs from 'fs'
import path from 'path'

export default (req, res) => {
  const fileContent = fs.readFileSync(
    path.join(__dirname, '..', '..', 'package.json'), 
    'utf8'
  )
  // ...
}

Note: I know you can cheat the above example ☝️ with require, but that's not the point. 😄

[amytych]

Just wanted to second that, trying to implement file uploading using API routes. I can get the file to upload but then need to be able to access it again to upload it to S3 bucket.

[ScottSmith95]

I second this! Also, being able to read directories is very important for my company's usage as we keep our data like team members and blog posts in a content directory so we're looking for a way to require all files in the directory.

[Timer]

The above PR will fix this! ☝️ 🙏

[marlonmarcello]

How about fs.writeFile is that possible? For example, create and save a JSON file based on a webhook that was posted on an /api/route

[huv1k]

Hey @marlonmarcello, this is going to be possible. Stay tuned 😊

[NicolasHz]

It's this already solved?

[huv1k]

Not yet, you can subscribe for #8334

[NicolasHz]

@huv1k Many thanks!

[BrunoBernardino]

Is there a way to help this move forward more quickly?

[bitjson]

Worth noting: if you're using TypeScript, you can already import a JSON file as a module directly (make sure resolveJsonModule is true in tsconfig.json). E.g.:

import myJson from '../../../some/path/my.json';

The shape of the JSON object is also automatically used as its type, so autocomplete is really nice.

[jkjustjoshing]

Workaround I'm using:

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

and in the location you need the path

import fs from 'fs'
import path from 'path'
import getConfig from 'next/config'
const { serverRuntimeConfig } = getConfig()

fs.readFile(path.join(serverRuntimeConfig.PROJECT_ROOT, './path/to/file.json'))

I know this doesn't solve the need to reference files with paths relative to the current file, but this solves my very related use case (reading image files from a /public/images folder).

[IanMitchell]

Saw in the PR this has changed a bit - any update on what the current plans are (or aren't)? Sounds like there are some strategies you don't want pursued, mind listing them + why so contributors can give this a shot?

[timsuchanek]

This is blocking the usage of nexus with Next.js. It would be great to see this prioritized again.

[karthhhi]

Workaround I'm using:

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

and in the location you need the path

import fs from 'fs'
import path from 'path'
import getConfig from 'next/config'
const { serverRuntimeConfig } = getConfig()

fs.readFile(path.join(serverRuntimeConfig.PROJECT_ROOT, './path/to/file.json'))

I know this doesn't solve the need to reference files with paths relative to the current file, but this solves my very related use case (reading image files from a /public/images folder).

Awesome Man. Worked for me.

[ScottSmith95]

I've been using the new getStaticProps method for this (in #9524). The method is currently marked as unstable but there seems to be good support from the Next.js team on shipping it officially.

e.g.:

export async function unstable_getStaticProps() {
  const siteData = await import("../data/pages/siteData.json");
  const home = await import("../data/pages/home.json");

  return {
    props: { siteData, home }
  };
}

[Svish]

@ScottSmith95 Do you have some public source project where you're using this? Curious about what it would look like.

[ScottSmith95]

The project is not open source, yet but I am happy to share more of my config if you have more questions.

[Svish]

@ScottSmith95 I have all the questions 😛

1. Where in your project do you store the data-files? (outside/inside src?)
2. What does a next.js page component using them look like?
3. Is it hard-coded paths only, or can you load a file based on path parameters?
4. How does build/deployment work, especially if it's not hard-coded paths?

[ScottSmith95]

@Svish We store data files in /data within our project. (Pages are in /pages, not /src/prages.) This page component looks like this (props are sent to the Home component which is the default export):

// /pages/index.js
const Home = ({ siteData, home }) => {
  return (
    <>
      <Head>
        <meta name="description" content={siteData.siteDescription} />
        <meta name="og:description" content={siteData.siteDescription} />
        <meta
          name="og:image"
          content={getAbsoluteUrl(siteData.siteImage, constants.siteMeta.url)}
        />
      </Head>
      <section className={`container--fluid ${styles.hero}`}>
        <SectionHeader section={home.hero} heading="1">
          <div className="col-xs-12">
            <PrimaryLink
              href={home.hero.action.path}
              className={styles.heroAction}
            >
              {home.hero.action.text}
            </PrimaryLink>
          </div>
        </SectionHeader>
        <div className={styles.imageGradientOverlay}>
          <img src={home.hero.image.src} alt={home.hero.image.alt} />
        </div>
      </section>
    </>
  );
};

For more advanced pages, those with dynamic routes, we grab this data like so:

// /pages/studio/[member.js]
export async function unstable_getStaticProps({ params }) {
  const siteData = await import("../../data/pages/siteData.json");
  const member = await import(`../../data/team/${params.member}.json`);

  return {
    props: { siteData, member }
  };
}

Deployment goes really smoothly, with dynamic routes, getStaticPaths() becomes necessary. I encourage you to check out the RFC for the documentation on that, but here's an example of how we handle that by gathering all our team member data and passing it to Next.js.

// /pages/studio/[member.js]
export async function unstable_getStaticPaths() {
  const getSingleFileJson = async path => await import(`../../${path}`);

  // These utility functions come from `@asmallstudio/tinyutil` https://github.com/asmallstudio/tinyutil
  const directoryData = await getDirectory(
    "./data/team",
    ".json",
    getSingleFileJson,
    createSlugFromTitle
  );
  const directoryPaths = directoryData.reduce((pathsAccumulator, page) => {
    pathsAccumulator.push({
      params: {
        member: page.slug
      }
    });

    return pathsAccumulator;
  }, []);

  return directoryPaths;
}

[Svish]

@ScottSmith95 Looks promising! A couple of follow-up questions if you have time:

1. What you're doing here is for static site generation? I.e. when using next export?
2. Have I understood it correctly, that getStaticPaths returns a list of path parameters, which is then (by next) fed, one by one, into getStaticProps for each render?
3. Can you use getStaticProps without getStaticPaths, for example for a page without any parameters?
4. Can you use getStaticProps in _app? For example if you have some site wide config you'd like to load or something like that?

[timneutkens]

@svish https://github.com/zeit/next.js/issues/9524

[NicolasHz]

What about the apis?? Those hooks are for pages, but what about apis?

[josias-r]

I'm confused. I was able to set the _dirname as an env variable in the next config. Therefore I was able to access the filesystem from the API, but it only worked locally. After deploying it to now, I got an error. Any ideas why it won't work after deployment?

[BrunoBernardino]

@josias-r the main issue is usually that the files to be read are not included the deployment, but it depends on how you include them and which types of files they are (js/json is usually fine, but other file types like .jade will require alternative ways of dealing with his, like using a separate @now/node lambda/deployment for reading/handling those files).

If you can explain more about the error, maybe someone can help you.

[josias-r]

@BrunoBernardino It was actually referring to JSON files inside my src folder. But it's actually even the fs.readdirSync(my_dirname_env_var) method that already fails in deployment. So that dir doesn't seem to exist at all after deployment. Here is what I get when I try to access the full path to the json vis my API:

ERROR   Error: ENOENT: no such file or directory, open '/zeit/3fc37db3/src/content/somejsonfilethatexists.json'

And as I mentioned, this works locally when I build and then run npm start.

[BrunoBernardino]

@josias-r Thanks! Have you tried doing the fs.readdirSync with a relative path (no variables) instead (just to debug the deployment)? I've found that to usually work, and if so, you can write that piece of code (just reading the file, not storing it anywhere) somewhere in an initialization process (getInitialProps or something), so that the deployment process picks up that it needs that file, and then keep reading it with the var in the actual code/logic. It's not neat, but it works until this is supported. I believe that also using __dirname works in some cases.

[josias-r]

@BrunoBernardino I was able to build a file tree starting from the root-relative path ./. What I got was the following JSON (without the node modules listed):

{
  "path": "./",
  "name": ".",
  "type": "folder",
  "children": [
    {
      "path": ".//.next",
      "name": ".next",
      "type": "folder",
      "children": [
        {
          "path": ".//.next/serverless",
          "name": "serverless",
          "type": "folder",
          "children": [
            {
              "path": ".//.next/serverless/pages",
              "name": "pages",
              "type": "folder",
              "children": [
                {
                  "path": ".//.next/serverless/pages/api",
                  "name": "api",
                  "type": "folder",
                  "children": [
                    {
                      "path": ".//.next/serverless/pages/api/posts",
                      "name": "posts",
                      "type": "folder",
                      "children": [
                        {
                          "path": ".//.next/serverless/pages/api/posts/[...id].js",
                          "name": "[...id].js",
                          "type": "file"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "path": ".//node_modules",
      "name": "node_modules",
      "type": "folder",
      "children": ["alot of children here ofc"]
    },
    { "path": ".//now__bridge.js", "name": "now__bridge.js", "type": "file" },
    {
      "path": ".//now__launcher.js",
      "name": "now__launcher.js",
      "type": "file"
    }
  ]
}

[BrunoBernardino]

Your JSON file seems to be missing there, did you try including it via the code like I suggested above? The main problem is that the optimizations the deployment runs don’t always pick up dynamic paths, I believe, so forcing a static path has worked for me in the past (not necessarily for the actual code running, but to make sure the relevant files are included). Does that makes sense?

[josias-r]

@BrunoBernardino I've switched to a non API solution. Since I dynamically want to require files from a folder and I only need the content of these files, I'm able to use the import() method. I just didn't want to do it this way, because it seems hacky, but it's essentially doing the same thing my API endpoint would have done. ... I tried putting the file into the static folder but that didn't work either. But I hope accessing the filesystem will be possible in the future.

[BrunoBernardino]

I've also had to resort to hacky solutions, but hopefully this will land soon and more people will start seeing Next as production-ready as these use cases become supported "as expected".

[paul-vd]

Workaround I'm using:

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

and in the location you need the path

import fs from 'fs'
import path from 'path'
import getConfig from 'next/config'
const { serverRuntimeConfig } = getConfig()

fs.readFile(path.join(serverRuntimeConfig.PROJECT_ROOT, './path/to/file.json'))

I know this doesn't solve the need to reference files with paths relative to the current file, but this solves my very related use case (reading image files from a /public/images folder).

Awesome Man. Worked for me.

It works perfectly on local development, though it does not seem to work when deploying to now.

ENOENT: no such file or directory, open '/zeit/41c233e5/public/images/my-image.png'
    at Object.openSync (fs.js:440:3)
    at Object.readFileSync (fs.js:342:35)
    at getEmailImage (/var/task/.next/serverless/pages/api/contact/demo.js:123:52)
    at module.exports.7gUS.__webpack_exports__.default (/var/task/.next/serverless/pages/api/contact/demo.js:419:87)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:42:9) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/zeit/41c233e5/public/images/my-image.png'
}

I understand that the public folder gets moved to the route so I tried to force it to search in the base folder when on production but still got the same result:

ENOENT: no such file or directory, open '/zeit/5fed13e9/images/my-image.png'
    at Object.openSync (fs.js:440:3)
    at Object.readFileSync (fs.js:342:35)
    at getEmailImage (/var/task/.next/serverless/pages/api/contact/demo.js:124:52)
    at module.exports.7gUS.__webpack_exports__.default (/var/task/.next/serverless/pages/api/contact/demo.js:331:87)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:42:9) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/zeit/5fed13e9/images/my-image.png'
}

[BrunoBernardino]

@PaulPCIO the problem you're experiencing there is because it's not a .json, .js, or .ts file. The files under /public are "deployed" to a CDN but not to the lambda (AFAIK), so for that case you need either a dedicated lambda (@now/node) deployment with includeFiles, or, if you only need that single file, convert it to base64 and use that as a var (in a dedicated file or not).

[paul-vd]

Thanks @BrunoBernardino expected as much, I will use the base64 method

[NicolasHz]

It's some resolution to the __dirname in deployed environment??

[BrunoBernardino]

@NicolasHz can you elaborate? I didn’t quite understand your question.

[josias-r]

@BrunoBernardino Looking at the last comments, including mine, I'm pretty sure that the "map _dirname in the next config" hack doesn't work in deployment. Even w/ js and JSON files. At least for now deployment, that doesn't count for custom deployments probably.

[NicolasHz]

@BrunoBernardino I'm not able to use some variables poiting to the local path on the deployed env. __dirname it's undefined once deployed, and I'm unable to read a file from my apis scripts.

[BrunoBernardino]

Got it @NicolasHz . Yeah, you'll need to resort to one of the solutions above, depending on which kind of file you need to read/access.

[BrunoQuaresma]

Just confirming, the config.js is not working on deployments.

[sjcodebook]

Workaround I'm using:

# next.config.js
module.exports = {
  env: {
    PROJECT_DIRNAME: __dirname,
  },
}

and in the api definition where i need the path(allPosts folder contains all blogs in markdown format and it is located in project root )

import fs from 'fs'
import { join } from 'path'

const postsDirectory = join(process.env.PROJECT_DIRNAME, 'allPosts')

It is working perfectly on local development. But it is giving this error when deploying to zeit now.

[POST] /api/postsApi
11:00:13:67
Status:
500
Duration:
8.1ms
Memory Used:
76 MB
ID:
kxq8t-1585546213659-5c3393750f30
User Agent:
axios/0.19.2
{
  fields: [ 'title', 'date', 'slug', 'author', 'coverImage', 'excerpt' ],
  page: 1
}
2020-03-30T05:30:13.688Z    572075eb-4a7a-47de-be16-072a9f7005f7    ERROR   Error: ENOENT: no such file or directory, scandir '/zeit/1cc63678/allPosts'
    at Object.readdirSync (fs.js:871:3)
    at getPostSlugs (/var/task/.next/serverless/pages/api/postsApi.js:306:52)
    at module.exports.fZHd.__webpack_exports__.default (/var/task/.next/serverless/pages/api/postsApi.js:253:86)
    at apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:48:15)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  errno: -2,
  syscall: 'scandir',
  code: 'ENOENT',
  path: '/zeit/1cc63678/allPosts'
}

[BrunoBernardino]

@sjcodebook like @BrunoQuaresma said, that workaround only works locally. I'm still using a separate @now/node deployment for lambdas to access the filesystem, and call that file via a request from the app itself (or generate whatever static result I need before deploying). Kinda insane, but it works.

[valse]

Hi @BrunoBernardino... Do you mean a separate project with a custom node server?

However I don't understand why there's an "includeFiles" setting if then it's impossible to access them 🤔

[BrunoBernardino]

@valse it can be on the same project. Here's a snippet of my now.json:

{
  "builds": [
    {
      "src": "next.config.js",
      "use": "@now/next"
    },
    {
      "src": "lambdas/**/*.ts",
      "use": "@now/node",
      "config": {
        "includeFiles": ["email-templates/**"]
      }
    }
  ],
  "routes": [
    {
      "src": "/lambdas/(.+)",
      "dest": "/lambdas/$1.ts"
    }
  ]
}

That way I can call them via something like:

await ky.post(`${hostUrl}/lambdas/email?token=${someToken}`);

from inside a next api page, assuming I have a lambdas/email.ts file which handles sending emails and reading from template files like pug.

I hope that helps!

[BrunoBernardino]

Also, "includeFiles" only works for @now/node (maybe others, but not @now/next)

[talentlessguy]

@BrunoBernardino looks like if using node functions, it now can't read ESM!

this is what happens when I try to import a list of mdx pages:

code

import { NextApiRequest, NextApiResponse } from 'next'
import { promises as fs } from 'fs'
import { join } from 'path'
const { readdir } = fs

export default async (req: NextApiRequest, res: NextApiResponse) => {
  const postFiles = await readdir(join(process.cwd(), 'pages', 'blog'))

  const postNames: string[] = postFiles.filter((page: string) => page !== 'index.tsx')

  const posts = []

  for (const post of postNames) {
    const mod = await import(`../pages/blog/${post}`)

    posts.push({ ...mod, link: post.slice(0, post.indexOf('.')) })
  }

  res.status(200).json([])
}

the error I get:

export const title = 'My new website!'
^^^^^^

SyntaxError: Unexpected token 'export'

[BrunoBernardino]

@talentlessguy I'm not on the Zeit/Vercel team, just a happy customer. Seems like that might be better suited to their customer support, as I see a few potential issues just from that snippet:

1. You might want to use just nothing or __dirname instead of process.cwd() for base path. I haven't used the latter in lambdas, but the others, so I'm not sure if that's an issue or not
2. You're importing NextApiRequest and NextApiResponse as types, but this should be running from @now/node", right? So the types should be imported like:

import { NowRequest, NowResponse } from '@now/node';

3. You're importing/reading from pages/... but are you including them via includeFiles? What does your now.json look like?

[talentlessguy]

@BrunoBernardino

I can't use __dirname because it is always /, process.cwd() instead, shows the real path

I accepted ur fixes and it worked:

lambdas/posts.ts

import { NowResponse, NowRequest } from '@now/node'
import { promises as fs } from 'fs'
import { join } from 'path'
const { readdir } = fs

export default async (req: NowRequest, res: NowResponse) => {
  const postFiles = await readdir(join(process.cwd(), 'pages', 'blog'))

  const postNames: string[] = postFiles.filter((page: string) => page !== 'index.tsx')

  const posts = []

  for (const post of postNames) {
    const mod = await import(`../pages/blog/${post}`)

    posts.push({ ...mod, link: post.slice(0, post.indexOf('.')) })
  }

  res.status(200).json([])
}

now.json

{
  "builds": [
    {
      "src": "next.config.js",
      "use": "@now/next"
    },
    {
      "src": "lambdas/**/*.ts",
      "use": "@now/node",
      "config": {
        "includeFiles": ["pages/blog/*.mdx"]
      }
    }
  ],
  "routes": [
    {
      "src": "/lambdas/(.+)",
      "dest": "/lambdas/$1.ts"
    }
  ]
}

error I get:

import Meta from '../../components/Article/Meta.tsx'
^^^^^^

SyntaxError: Cannot use import statement outside a module

Looks like typescript node function can't treat .mdx as a module :(

[BrunoBernardino]

Alright, so it seems you found the problem. Try reading the file contents and parsing them instead of importing directly. I’ve never seen an import like that work, and it seems like something that would only work with some Babel magic, which you’re also welcome to use instead of plain TS.

[talentlessguy]

@BrunoBernardino you're right, but it's not plain ts... I have the target set to esnext and module to esnext also, it should be able to import everything... but somehow it doesn't

anyways it's not related to the issue, gonna google it somewhere

[BrunoBernardino]

No worries. A couple of tips might be in https://mdxjs.com/advanced/typescript and https://mdxjs.com/getting-started/webpack which might make it so the @now/node deployment needs to be tweaked to use it. Anyway, their support should be of help.