Closed Timer closed 2 years ago
any movement on this? It would be great to be able to include html email templates for use in API Routes. Right now I am including them in JS files but I am not a particular fan of this hack.
Another hack is to use webpack raw-loader to embed them into js.
yarn add --dev raw-loader
const templates = {
verify: require("raw-loader!../template/email/verify.hbs").default,
};
Then use templates.verify
as a string.
There's an issue going on with next-i18next that seems to be related to this one (vercel/vercel#4271) . Basically now
doesn't put the .json
files located inside /public/static/locales/
into the serverless function. Can anyone provide a workaround until the feature discussed here is added to next?
@borispoehland have you tried the import/require workarounds from above? That should work.
@borispoehland have you tried the import/require workarounds from above? That should work.
@BrunoBernardino I don't know what exact comment you mean.
Can you give me an example of somehow importing all the .json
files inside public/static/locales
into the serverless function? And where to do this (in what file)?
I'm using next (as you stated earlier, includeFiles
isn't compatible with @now/next
, idk if this has any impact on my problem).
Besides, because next-i18next
is kind of a blackbox to me (thus I don't want to import the files from there), I search for a way to entirely import them so that next-i18next
can directly access them (in other comments above, sometimes only the PROJECT_DIRNAME
was defined inside the next.config.json
and the import had to be done manually. This is not what I try to reach). Like in vercel/vercel#4271, I just want now
to take my .json
files into the serverless function somehow.
@borispoehland in any file inside pages/api
(or that gets called by one there), do something like https://github.com/vercel/next.js/issues/8251#issuecomment-544008976
You don't need to do anything with the import. The point is that the webpack vercel runs will then see those files need to be included, and it should work.
I hope that makes sense.
@borispoehland in any file inside
pages/api
(or that gets called by one there), do something like #8251 (comment)You don't need to do anything with the import. The point is that the webpack vercel runs will then see those files need to be included, and it should work.
I hope that makes sense.
@BrunoBernardino the problem with this approach is that I have lots of json files. Doing the import manually for every file is kind of cumbersome. Is there a easier way to tell now
: "Hey, please pick up all json files inside that directory recursively"? Thanks in advance
Edit: Even manually importing json
files results in the same error than before. I'm going to open a new issue for this, I guess
I opened a new issue for my problem, in case someone is interested in joining the discussion. Thanks for now, @BrunoBernardino !
Another option / workaround to enable the ability to use __dirname
as you would normally expect it to behave is to adjust the webpack config.
By default, webpack will alias various Node globals with polyfills unless you tell it not to:
https://webpack.js.org/configuration/node/
And the webpack default settings are to leave __dirname
and __filename
alone, i.e. not polyfill them and let node handle them as normal.
However, the Next.js webpack config doesn't use / reflect the webpack defaults https://github.com/vercel/next.js/blob/bb6ae2648ddfb65a810edf6ff90a86201d52320c/packages/next/build/webpack-config.ts#L661-L663
All of that said, I have used the below custom Next config plugin to adjust the webpack config.
IMPORTANT: this works for my use case. It has not been tested in a wide range of environments / configurations nor has it been tested against all of the Next.js unit/integration tests. Using it may have unintended side-effects in your environment. Also, Next may have specific reasons for not using the webpack default settings for
__dirname
and__filename
. So again, the code below may have unintended side-effects and should be used with caution.
Also, the below plugin has been designed for use with the next-compose-plugins
package: https://github.com/cyrilwanner/next-compose-plugins
But should work as a normal plugin as well: https://nextjs.org/docs/api-reference/next.config.js/custom-webpack-config
const withCustomWebpack = (nextCfg) => {
return Object.assign({}, nextCfg, {
webpack(webpackConfig, options) {
// We only want to change the `server` webpack config.
if (options.isServer) {
// set `__dirname: false` and/or `__filename: false` here to align with webpack defaults:
// https://webpack.js.org/configuration/node/
Object.assign(webpackConfig.node, { __dirname: false });
}
if (typeof nextCfg.webpack === 'function') {
return nextCfg.webpack(webpackConfig, options);
}
return webpackConfig;
},
});
};
I implemented the solution by @jkjustjoshing, and while it works great locally, it does not work when I deploy the app to Vercel.
I get the following error:
Error: GraphQL error: ENOENT: no such file or directory, open '/vercel/37166432/public/ts-data.csv'
My code:
const content = await fs.readFile(
path.join(serverRuntimeConfig.PROJECT_ROOT, "./public/ts-data.csv")
);
Here's a link to the file: https://github.com/bengrunfeld/trend-viewer/blob/master/pages/api/graphql-data.js
@bengrunfeld yes, your solution only works locally.
I had a similar problem lately (wanted to read a file in a API route) and the solution was easier than expected.
Try path.resolve('./public/ts-data.csv')
@borispoehland Thank you SO much!! Your solution worked beautifully!
@bengrunfeld no problem, I also found it out on coincidence (@BrunoBernardino ;)). It's the solution to everyone's problem here, I guess.
Please note, you still need to set next.config.js
. I deleted the file after I saw that @borispoehland's solution worked, and received a similar error.
Then I reset it to @jkjustjoshing's solution above, deployed again to Vercel, and it worked.
# next.config.js
module.exports = {
serverRuntimeConfig: {
PROJECT_ROOT: __dirname
}
}
Please note, you still need to set
next.config.js
. I removed it after I saw that @borispoehland's solution worked, and received a similar error.I reset it to @jkjustjoshing's solution above, deployed again to Vercel, and it worked.
# next.config.js module.exports = { serverRuntimeConfig: { PROJECT_ROOT: __dirname } }
@bengrunfeld Really? Maybe you're still using the PROJECT_ROOT
approach at another point in the code, because in my project it works without it. How does the error look like?
When deploying to Vercel, how do I write a readFile
in a Page that will work both in SSG and SSR/Preview mode?
Demo repository of where it does not work: https://github.com/mathdroid/blog-fs-demo
@mathdroid try moving readFile
and readdir
inside the getStaticProps
and getStaticPaths
functions, respectively, otherwise the code might run in the browser.
Importing fs
should be fine on top, though.
Let us know how that works.
@borispoehland Thanks for the wonderful solution. Did not expect path.resolve()
to /public
would work both locally and on Vercel :eyes:! You're my savior for the day. :+1:
@borispoehland i tried your solution inside a serverless functionbut still get: ENOENT: no such file or directory, open '/var/task/public/posts.json'
const postsFile = resolve('./public/posts.json');
const updateCache = async (posts: IPost[]): Promise<IPost[]> => {
postCache = posts;
fs.writeFileSync(postsFile, JSON.stringify(postCache)); // <====
return postCache;
}
i tried with our without the next.config.js
module.exports = {
serverRuntimeConfig: {
PROJECT_ROOT: __dirname
}
}
Maybe your solution does not work on serverless functions?
@borispoehland i tried your solution inside a serverless functionbut still get: ENOENT: no such file or directory, open '/var/task/public/posts.json'
const postsFile = resolve('./public/posts.json'); const updateCache = async (posts: IPost[]): Promise<IPost[]> => { postCache = posts; fs.writeFileSync(postsFile, JSON.stringify(postCache)); // <==== return postCache; }
i tried with our without the next.config.js
module.exports = { serverRuntimeConfig: { PROJECT_ROOT: __dirname } }
Maybe your solution does not work on serverless functions?
I don't know why it doesn't work on your end... Sorry
Ok i made it work for read using @bengrunfeld code but unfortunately you cannot write: [Error: EROFS: read-only file system, open '/var/task/public/posts.json'] So no way to update a cache to avoid too much database calls :(
@neckaros have you tried using my approach to read a file other than .json
, e.g. a .jpg
file?
Ok i made it work for read using @bengrunfeld code but unfortunately you cannot write:
[Error: EROFS: read-only file system, open '/var/task/public/posts.json']
So no way to update a cache to avoid too much database calls :(
@neckaros you should be able to write and read from S3 (or some other, external filesystem), but I usually use redis for quick, cached things that can be volatile. https://redislabs.com keeps it "serverless", and I've got production-ready code examples in https://nextjs-boilerplates.brn.sh if you want.
@borispoehland i could read but not write from serveless function. But I ended up having it working by refresh my cache in the incremental builds (revalidate) instead of on add new content . Which I guess is not a bad pattern. Thanks for your help!
@BrunoBernardino thanks i will have a look. Really looking to a fully free hobbyist solution that does not break once you have a few users :)
Really looking to a fully free hobbyist solution that does not break once you have a few users :)
Copy that. RedisLabs and Vercel did that for me. 💯
After some digging I got writing files working with the extended os package...
import { tmpdir } from "os";
const doc = new PDFDocument()
const pdfPath = path.join(tmpdir(), `${store.id}${moment().format('YYYYMMDD')}.pdf`)
const writeStream = doc.pipe(fs.createWriteStream(pdfPath)
reading a file works with @subwaymatch solution
const logoPath = path.resolve('./public/logo.png')
After some digging I got writing files working with the extended os package...
import { tmpdir } from "os"; const doc = new PDFDocument() const pdfPath = path.join(tmpdir(), `${store.id}${moment().format('YYYYMMDD')}.pdf`) const writeStream = doc.pipe(fs.createWriteStream(pdfPath)
reading a file works with @subwaymatch solution
const logoPath = path.resolve('./public/logo.png')
Nice, are you able to read back the contents of this file? Is the directory accessible and permanent?
@marklundin With a function named tmpdir
I doubt it's permanent, but if this works, then it would be good to know how temporary actually tmpdir
is, yeah... 🤔
Any updates on this? I'm wondering why it works in getInitialProps, but not in API routes 🤷♂️
My current workaround
const data = await import(`../../../../data/de/my-nice-file.json`);
res.json(data.default);
currently having this issue in API routes too
currently having this issue in API routes too
There are a few working solutions here, what problem are you having, specifically?
I'm struggling to get this working even with suggestions from this thread. My use-case is I'm writing a guide and want to show the source-code for the component alongside the component itself. My method for doing this is using fs to load the component's jsx file inside getServerSideProps and passing the string value of the file contents as a prop.
I was feeling over the moon about having it working locally, but then when I went to deploy it, the joy has gone :(
Please see: https://github.com/ElGoorf/i18next-guide/blob/fix-example-components/pages/plurals.jsx
@ElGoorf your problem is that the public
files are on an edge, and the functions are on a lambda. Now, @vercel/next
still doesn't allow for includeFiles
, so the easiest way for you to get it working would be to use a lambda
function with it.
Here's some sample code that helped others here: https://github.com/vercel/next.js/issues/8251#issuecomment-614220305
Thanks @BrunoBernardino I didn't realise I'd missed the "x hidden items load more..." and thought I was going crazy from the thread losing meaning!
Unfortunately, I struggled with your solution, as it's the first time I've heard of Edge/Lambda, however, I found @balthild's solution was closer to what I was originally after before trying the node.fs method: https://github.com/vercel/next.js/issues/8251#issuecomment-634829189
Great! Did you get it working? Or are you still having issues?
I'm not sure Vercel even uses that terminology, but by Edge I mean CDN, where static files are served from, and by lambda I mean the "backend" functions that get called from the API routes, which are isolated like AWS Lambda functions.
Hey,
Any update on writing to files using next.js on vercel? I can read no problem. Using the const logoPath = path.resolve('./public/logo.png')
I'm attempting to overwrite the public/sitemap.xml file (due to the size limits on vercel) I can only return it without error as a static file in the public folder. I have previously implemented the sitemap with zlib and streaming the response but it seems to wait until the stream is finished and then return it. This doesn't hit the size limitation error, but unfortunately it's very slow. I'm open to any suggestions people might have. The sitemap is built from an API call to a separate backend and needs to be updated regularly.
Things I have attempted :
Hey @emomooney, I don't imagine Vercel ever allowing to write files in a function (even for caching), since the main "advantage" of serverless is its statelessness, and that would add state to it, so I think you'll need to use the edge/cdn for it.
I have previously implemented the sitemap with zlib and streaming the response but it seems to wait until the stream is finished and then return it.
I'm curious if you were just experiencing this slowness for subsequent calls, or just the first, for the cold start? I imagine this was an API call to Vercel via a next.js api function, or a dedicated lambda, similar to what I do here.
If it was and it was still too slow, is your "separate backend" outside of Vercel? If so, you can potentially use it to build a sitemap.xml
file and vercel --prod
it into a domain, basically "caching" the file to be readable and accessible, and you'd just need to update the robots.txt
to link the sitemap to another domain/subdomain.
Also just ran into this issue, and this one becomes extra annoying when your next app is part of a monorepo setup, since ./
can mean very different things depending on the environment you're running the API handler in.
I'm trying to read a directory (pages
directory) in an API route.
When I try @jkjustjoshing's workaround and do readdirSync(join(serverRuntimeConfig.PROJECT_ROOT, './pages'))
, I get this error in Vercel Functions realtime logs:
2020-12-23T16:05:16.924Z 9292cb9e-55fa-4f62-af7e-416dbd7b81cc ERROR Error: ENOENT: no such file or directory, scandir '/vercel/workpath0/pages'
at readdirSync (fs.js:955:3)
at Module.IPik (/var/task/.next/serverless/pages/api/testFunc.js:227:58)
at __webpack_require__ (/var/task/.next/serverless/pages/api/testFunc.js:23:31)
at module.exports.OjAL.__webpack_exports__.default (/var/task/.next/serverless/pages/api/testFunc.js:485:34) {
errno: -2,
syscall: 'scandir',
code: 'ENOENT',
path: '/vercel/workpath0/pages'
}
2020-12-23T16:05:16.925Z 9292cb9e-55fa-4f62-af7e-416dbd7b81cc ERROR Unhandled Promise Rejection {"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"Error: ENOENT: no such file or directory, scandir '/vercel/workpath0/pages'","reason":{"errorType":"Error","errorMessage":"ENOENT: no such file or directory, scandir '/vercel/workpath0/pages'","code":"ENOENT","errno":-2,"syscall":"scandir","path":"/vercel/workpath0/pages","stack":["Error: ENOENT: no such file or directory, scandir '/vercel/workpath0/pages'"," at readdirSync (fs.js:955:3)"," at Module.IPik (/var/task/.next/serverless/pages/api/testFunc.js:227:58)"," at __webpack_require__ (/var/task/.next/serverless/pages/api/testFunc.js:23:31)"," at module.exports.OjAL.__webpack_exports__.default (/var/task/.next/serverless/pages/api/testFunc.js:485:34)"]},"promise":{},"stack":["Runtime.UnhandledPromiseRejection: Error: ENOENT: no such file or directory, scandir '/vercel/workpath0/pages'"," at process.<anonymous> (/var/runtime/index.js:35:15)"," at process.emit (events.js:326:22)"," at processPromiseRejections (internal/process/promises.js:209:33)"," at processTicksAndRejections (internal/process/task_queues.js:98:32)"]}
Unknown application error occurred
Which presumably doesn't work because those files aren't being copied into the serverless environment?
Is there a way to do this?
@remjx it's not currently possible. What are you trying to do? Is it something you can do before deploying? Or can it be deployed as a separate @vercel/node
function/build?
@BrunoBernardino I have something like this:
/pages/api/createUser.js
/pages/[userName].jsx
/pages/blog/...
I don't want a User to be able to be created with the name "blog" because it conflicts with an existing route. So I'd like to be able to read the pages
directory so I can automatically add "blog" to the list of off-limits usernames.
Is it something you can do before deploying?
Good idea. I'm now generating a .json file in the /public
folder during build but am still getting an ENOENT error when I try to read the file.
I don't understand what's going on in your workaround to read files. Is there documentation for these configuration options?
@BrunoBernardino I have something like this:
/pages/api/createUser.js /pages/[userName].jsx /pages/blog/...
I don't want a User to be able to be created with the name "blog" because it conflicts with an existing route. So I'd like to be able to read the
pages
directory so I can automatically add "blog" to the list of off-limits usernames.Is it something you can do before deploying?
Good idea. I'm now generating a .json file in the
/public
folder during build but am still getting an ENOENT error when I try to read the file.I don't understand what's going on in your workaround to read files. Is there documentation for these configuration options?
I know this isn't exactly what you'd like to achieve, but have you thought about prepending a "user" in front of the route?
/pages/user/[userName].jsx
That way, you don't have to worry about conflicting usernames, especially not when new pages are added in the future
Edit: I thought about it for a short time and I'm sure now that you have to prepend at least something to the route. Otherwise, let's assume that your project is successful and you have many users. In the future, you then want to add a new page: /pages/about.jsx
. To prevent conflicts, that would mean that you possibly have to delete an existing user account of the user named "about". I don't think that you want this race condition to be possible to occur, so I think the smartest way is to prevent a conflict by prepending something in front of the slug.
@remjx
Good idea. I'm now generating a .json file in the /public folder during build but am still getting an ENOENT error when I try to read the file.
Yes, public/*
files are deployed to an edge/CDN, not available to the serverless function. You should be able to generate that .json
file inside the pages
directory and simply import it or require it (make sure your tsconfig.json
supports it, maybe also jest
if you're using it), and the file will be available.
I don't understand what's going on in your workaround to read files. Is there documentation for these configuration options?
Yes, but it's always changing and it's "deprecated" now: https://vercel.com/docs/configuration#project/builds
@BrunoBernardino moving the .json file to /pages
and importing it works. Thank you!
Did anyone manage to be able to read from their filesystem in getServerSideProps
? I tried @jkjustjoshing's suggestion, but it only works locally and not when deployed to Vercel.
Reproducible example: https://github.com/robertcoopercode/with-mdx-remote-app
Related GitHub discussion: https://github.com/vercel/next.js/discussions/22853
@robertcoopercode Maybe this code example helps (it used to be d deployed to Vercel, though now it's not). I think you need __dirname
or process.cwd()
Just noticed this thread and wanted to share a short note.
I've been able to read successfully files located in the public
dir from serverless functions using fs.readFile
/fs.readDir
:
fs.readdirSync(path.resolve("./public", "company-logos"));
Today I noticed that enabling Webpack 5 breaks this behaviour (no clue why), causing the function to throw:
ERROR Error: ENOENT: no such file or directory, scandir '/var/task/packages/website/public/company-logos'
@robertcoopercode how did you manage to solve this issue? I am accessing the filesystem using getServerSideProps
and process.cwd()
which works fine locally but it is still failing on Vercel for me with the following error:
2021-04-28T07:38:22.294Z 060b539a-3bb0-4c61-b009-4870d16fbce4 ERROR Unhandled error during request: Error: ENOENT: no such file or directory, open '/var/task/content/page.md'
at Object.openSync (fs.js:476:3)
at Object.readFileSync (fs.js:377:35)
@fernandoabolafio my code example from above worked in Vercel up until very recently (I'm no longer deploying to it, but you should be able to clone and deploy to test). If that no longer works, maybe they changed something.
@BrunoBernardino thanks for your answer. I just found this: https://vercel.com/support/articles/how-can-i-use-files-in-serverless-functions. It is an official note from Vercel saying that accessing the filesystem from pages SSR isn't working. It is actually pointing to this thread as a reference.
Feature request
Is your feature request related to a problem? Please describe.
It's currently not possible to read files from API routes or pages.
Describe the solution you'd like
I want to be able to call
fs.readFile
with a__dirname
path and have it "just work".This should work in Development and Production mode.
Describe alternatives you've considered
This may need to integrate with
@zeit/webpack-asset-relocator-loader
in some capacity. This plugin handles these types of requires.However, it's not a necessity. I'd be OK with something that only works with
__dirname
and__filename
(no relative or cwd-based paths).Additional context
Example: