Javascript Out of Memory Heap Error when 1,900+ Posts Added

[iChris]

Relative newb here so forgive me if my terminology isn't quite correct. I'll do my best to clarify where needed:

Describe the bug I'm getting this error when I try to run Eleventy after putting all my archived blog posts into /src/posts/. It runs fine with ~10 posts.

To Reproduce Dump a whole bunch of .md files into the Posts folder.

Expected behavior Not crash.

Screenshots

• npm run error screen
• npm debug log

Environment:

• OS and Version: macOS 10.14.6
• Eleventy Version 0.7.1

Additional context

I'm not sure if it's happening because it's trying to build the /tags/ out - I'd be fine losing those if need be.

Thanks in advance!

[coolsoftwaretyler]

It might be helpful to take a look at your project. I'm getting varying results using the Eleventy Base Blog with 2000 copied posts.

Steps I took to reproduce

1. Clone https://github.com/11ty/eleventy-base-blog
2. Make 2000 copies of /posts/post1.md
3. Run npx eleventy

v0.9.0

Success. Output:

Copied 2 items and Processed 2116 files in 5.19 seconds (2.5ms each, v0.9.0)

v0.7.1

Failure. But a different error message that looks related:

Problem writing Eleventy templates: (more in DEBUG output)
> Having trouble rendering njk template ./feed/feed.njk (TemplateContentRenderError)
> (unknown path)
  RangeError: Maximum call stack size exceeded (Template render error):
    Template render error: (unknown path)
      RangeError: Maximum call stack size exceeded
        at Object._prettifyError (/Users/tylerwilliams/ogden/eleventy-base-blog/node_modules/nunjucks/src/lib.js:36:11)
        at /Users/tylerwilliams/ogden/eleventy-base-blog/node_modules/nunjucks/src/environment.js:567:19
        at Template.root [as rootRenderFunc] (eval at _compile (/Users/tylerwilliams/ogden/eleventy-base-blog/node_modules/nunjucks/src/environment.js:637:18), <anonymous>:79:3)
        at Template.render (/Users/tylerwilliams/ogden/eleventy-base-blog/node_modules/nunjucks/src/environment.js:556:10)
        at /Users/tylerwilliams/ogden/eleventy-base-blog/node_modules/@11ty/eleventy/src/Engines/Nunjucks.js:133:14
        at new Promise (<anonymous>)
        at /Users/tylerwilliams/ogden/eleventy-base-blog/node_modules/@11ty/eleventy/src/Engines/Nunjucks.js:132:14
        at Template.render (/Users/tylerwilliams/ogden/eleventy-base-blog/node_modules/@11ty/eleventy/src/TemplateContent.js:167:28)
        at processTicksAndRejections (internal/process/task_queues.js:89:5)
        at async Template._getContent (/Users/tylerwilliams/ogden/eleventy-base-blog/node_modules/@11ty/eleventy/src/Template.js:450:15)
Copied 2 items and Processed 0 files in 1.64 seconds

Removing problem file from 0.7.1

I removed ./feed/feed.njk and ran npx eleventy again. It worked, giving me this:

Copied 2 items and Processed 2115 files in 5.20 seconds (2.5ms each)

It looks like Eleventy can handle 2000 posts, but perhaps some templating engine or other part of the process is choking?

[iChris]

Oh maybe my first step would be updating Eleventy? I'm not sure how to do that. Currently the how to upgrade docs issue is awaiting more votes. :)

The site is just my attempt at moving my WordPress site over to Eleventy / Netlify. Repo here which has all the blog posts added right now on this branch.

If there's a quick way for me to upgrade, I'll happily try that as there's nothing I'm worried about losing nor is this a live site yet. Thanks!

[coolsoftwaretyler]

So I cloned your repo, switched to the adding-blog-posts branch, ran npm i and npm start and the site built and worked, serving locally.

But it took about 92 seconds, and definitely slowed down around the tags generation. I wonder if Netlify has a memory limit and you're hitting it at that point?

Edit

No problem with npm run production locally, either. 82 seconds this go around.

[iChris]

Thanks for testing it out. I was getting the error locally on my iMac when I submitted the issue.

But I was actually just testing it here on my laptop and it worked so I'm not sure what's up. I'll have to re-run it when I'm back at the office tomorrow and see if it was me or just running it on Mondays. ;)

The tag generation part I could definitely live without until I've had a chance to clean up the posts.

[iChris]

Quick follow up - local version of Eleventy on my laptop is 0.8.3 (vs 0.7.1 on my iMac) so maybe that's what fixed it. I'll report back tomorrow.

[zachleat]

Ah, I have run into this locally when I was working on my Twitter Archive project (one pagination template generated over 27000 pages).

I was able to workaround it by using node’s --max-old-space-size param to allocate more memory to node, e.g.:

node --max-old-space-size=6000 ./node_modules/@11ty/eleventy/cmd.js --quiet

I filed https://github.com/11ty/eleventy/issues/698 related to this but let me know if the workaround also works for you

[zachleat]

(Upgrade instructions are included with release notes https://github.com/11ty/eleventy/releases/tag/v0.9.0)

Install to your Local project (preferred): npm install @11ty/eleventy
Already installed in your local project? Upgrade your version: npm update @11ty/eleventy
Want to install globally? npm install -g @11ty/eleventy
Already installed globally? Upgrade your Global version: npm update -g @11ty/eleventy

[iChris]

(Upgrade instructions are included with release notes https://github.com/11ty/eleventy/releases/tag/v0.9.0)

Install to your Local project (preferred): npm install @11ty/eleventy
Already installed in your local project? Upgrade your version: npm update @11ty/eleventy
Want to install globally? npm install -g @11ty/eleventy
Already installed globally? Upgrade your Global version: npm update -g @11ty/eleventy

Thanks for pointing that out! I should've looked closer.

I'm still stuck on 0.7.1 somehow? Screenshot I've tried updating globally and locally and nothing pushes it to 0.9.0. (If I should open a new issue, let me know and I'll do so.)

Thanks! Chris.

[coolsoftwaretyler]

Did you update your local or global version of Eleventy? You're running eleventy --version against the global package. Check the package.json in your project and compare.

[coolsoftwaretyler]

Check the difference between local/global installation in the docs here:

https://www.11ty.io/docs/usage/

[iChris]

Did you update your local or global version of Eleventy? You're running eleventy --version against the global package. Check the package.json in your project and compare.

The package.json in the project lists 0.8.3. screenshot Now I'm really confused. :)

[coolsoftwaretyler]

You can install node packages, like Eleventy, globally and/or locally.

When you run npm install PACKAGENAME -g, it gets installed globally and usually aliased in your command line. So at some point, if you've run npm install @11ty/eleventy -g, your command line seeks out the global installation when you just run eleventy in the command line.

Here, you're running eleventy --version, and it's likely looking to the version of the global Eleventy you have.

But in your package.json, you've updated to 0.8.3. When you run npm install inside the project directory, the local installation will be set to 0.8.3.

In the CLI usage doc I previously linked, there are two examples. One runs the local installation of eleventy, with npx @11ty/eleventy. The other runs the global installation of eleventy, with eleventy.

Here's more about global installation/local installation and why you should use local instead of global: https://www.11ty.io/docs/global-installation/

My guess is you have updated your local install correctly, but have not updated your global install. Looking at your project, I think the boilerplate npm commands like npm run production should be using the local version, so you're probably fine.

If you want to thoroughly check that the local version is correct, remove your node_modules folder, remove the package-lock.json file, and then run npm install inside the project again.

[iChris]

Thanks @ogdenstudios. I think I kind of follow most of that. :)

If you want to thoroughly check that the local version is correct, remove your node_modules folder, remove the package-lock.json file, and then run npm install inside the project again.

Ok so I did this + I updated package.json to "@11ty/eleventy": "^0.9.0", - is that the proper way to do it? And then ran npm update @11ty/eleventy and got:

So I think I'm ok now?

Eleventy still crashes on load on my Desktop (error log here) but it's likely something I'm doing since it works on my laptop. I'll keep digging and try to report back if I get it working here. Thanks again for all your help @ogdenstudios & @zachleat.

Chris.

[coolsoftwaretyler]

No need to manually update your package.json. Running npm update @11ty/eleventy inside the project folder should do it for you neatly.

On the desktop, have you tried running @zachleat 's suggestion:

node --max-old-space-size=6000 ./node_modules/@11ty/eleventy/cmd.js --quiet

[iChris]

On the desktop, have you tried running @zachleat 's suggestion:

node --max-old-space-size=6000 ./node_modules/@11ty/eleventy/cmd.js --quiet

I'm not sure what it's supposed to do exactly but I don't get any more errors?

[iChris]

... and npm start still spits out an error (not sure if the above is supposed to fix that or not?)

Error log

[iChris]

Just for fun I tried moving all my posts over to a new repo with the Eleventy Base Blog starter kit and it seems to run fine - locally anyway. It's been building the site for 6+ minutes on Netlify over here :)

I should start a new career as "I'll break your open source app and have no idea how to help fix it and then just continue breaking things as I try and leave the room." $$$

[coolsoftwaretyler]

node --max-old-space-size will increase the max memory for Node on your machine and the second part is just running Eleventy afterwards.

If npm start is getting an error, my guess is it probably still needs to have the memory adjusted. You might want to adjust the npm start script to run with that memory flag.

Looking at the first repo you posted, looks like your start script is:

 "start": "concurrently 'npm run sass:process -- --watch' 'npm run cms:bundle -- --watch' 'chokidar \"src/_includes/**\" -c \"npm run cms:precompile\"' 'npm run serve'",

Which you could change to:

 "start": " node --max-old-space-size=6000 concurrently 'npm run sass:process -- --watch' 'npm run cms:bundle -- --watch' 'chokidar \"src/_includes/**\" -c \"npm run cms:precompile\"' 'npm run serve'",

And I think that might do it.

Or you could save the higher memory option as an environment variable like this article mentions.

[iChris]

This is all going a bit over my head - not that I'm going to stop banging away at things to try and figure it out but just not the easy replacement for a WordPress based hobby blog I was hoping for. :)

Is there any way to tell if the memory issue is because of the amount of posts (1,944) or because of something like the amount of tag pages Eleventy is having to generate? When I run npm start the last thing I see before it quits with an error is pages and pages of this:

Is there a flag or something I can set to tell Eleventy to ignore building pages for the tags?

[Ryuno-Ki]

@zachleat Ran into the situation of too many open files when importing my 37k tweets. graceful-fs fixed that for me.

[Ryuno-Ki]

@iChris Well, we're pushing the boundaries of a file-based system here. WordPress is working with databases, which are designed for cases like yours.

Perhaps you could stick around a bit longer. :+1: for asking if we got too technical.

[iChris]

Ran into the situation of too many open files when importing my 37k tweets. graceful-fs fixed that for me.

Haha that's the next phase - importing 80k tweets. ;) I actually moved them into my micro blog account so probably won't try and do that with Eleventy.

[iChris]

Well, we're pushing the boundaries of a file-based system here. WordPress is working with databases, which are designed for cases like yours.

Perhaps you could stick around a bit longer. 👍 for asking if we got too technical.

Oh I'm not going anywhere (yet). :) I did have that thought of maybe just starting fresh and slowly bringing the posts back as I have time to clean them up / etc. There's some google SEO juice I'd rather not lose but nothing mission critical.

How would I use graceful-fs in my situation?

[Ryuno-Ki]

@iChris Looking at https://github.com/11ty/eleventy/issues/695#issuecomment-533717072 I'd say, you don't. It is rather something @zachleat could consider adopting (e.g. for compiling the collections).

I have an account on micro.blog as well, but it didn't work out like I expected. In terms of IndieWeb I'd like to have control over my content (i.e. host it on my own and POSSE to micro.blog). My support request remains unanswered since months. But that's off-topic I guess. Feel free to drop me a line.

[zachleat]

@Ryuno-Ki how does graceful-fs compare to fs-extra? I use the latter in a few places in Eleventy internally.

[iChris]

In my case it may have been an out of date Node installation. I was setting up a new Mac and installed Node fresh and when I downloaded my repo and ran it, it was able to load it no problem.

Once I upgraded Node to v13.8.0 and set up a environment variable on Netlify to use a more current version of Node there, it was able to run no problem.

[chrissy-dev]

Ah, I have run into this locally when I was working on my Twitter Archive project (one pagination template generated over 27000 pages).

I was able to workaround it by using node’s --max-old-space-size param to allocate more memory to node, e.g.:

node --max-old-space-size=6000 ./node_modules/@11ty/eleventy/cmd.js --quiet

I filed #698 related to this but let me know if the workaround also works for you

Just want to note my own experience here.

I came across this same problem while trying to build a massive project with 11ty. The command suggested by @zachleat worked for me.

• 1 nunjucks template
• 66,912 markdown files, coming in at 272MB.

Output (501MB):

Copied 24 files / Wrote 66912 files in 332.96 seconds (5.0ms each, v0.11.0)

The use-case for this is quite bespoke but I'm impressed with that performance, we will be running the build once per month for archiving purposing. I can live with that 5 mins :)

[zachleat]

Wow, that’s incredible. But also that 5.0ms math is clearly wrong, right? 😅

Dunno what I was smoking when I thought that was wrong 🤦‍♂️

Filed #1399

[cfjedimaster]

Ah, I have run into this locally when I was working on my Twitter Archive project (one pagination template generated over 27000 pages). I was able to workaround it by using node’s --max-old-space-size param to allocate more memory to node, e.g.:

node --max-old-space-size=6000 ./node_modules/@11ty/eleventy/cmd.js --quiet

I filed #698 related to this but let me know if the workaround also works for you

Just want to note my own experience here.

I came across this same problem while trying to build a massive project with 11ty. The command suggested by @zachleat worked for me.

• 1 nunjucks template
• 66,912 markdown files, coming in at 272MB.

Output (501MB):

Copied 24 files / Wrote 66912 files in 332.96 seconds (5.0ms each, v0.11.0)

The use-case for this is quite bespoke but I'm impressed with that performance, we will be running the build once per month for archiving purposing. I can live with that 5 mins :)

I too am working with Twitter data. :) In my case, using Zach's suggestion worked for approx 40K files (I'm doing one file per tweet), but craps out above that. Is there no way to get Eleventy working in the 50-100K page range? I don't mind if it slows down.

[cfjedimaster]

I should add - I tried setting the size to 16000, and this does not give me a stack trace, but just a plain Killed.

[zachleat]

Just logging #2360 as a coordination issue for the “out of memory” pagination issue. Please follow along there!