Is Gatsby designed to support large (10,000+) article wikis?

[QuincyLarson]

Is Gatsby intended to allow wikis of 10,000s of articles? Is this a use case you intend to support?

Free Code Camp is having a bit of trouble getting the latest version to work with our current wiki of ~600 articles.

All the example wikis linked to in Gatsby's readme seem comparatively small.

And our wiki is growing quite quickly and may be 10k+ articles by the end of the year.

[KyleAMathews]

It is intended to work with very large sites of 10k+ articles but as you're perhaps the first to try creating a site this large, you're also the first to run into not-yet-resolved scaling problems :)

Could you describe in detail the problems you're hitting?

[QuincyLarson]

Awesome - I'm glad to hear this. I'm going to ask @berkeleytrue and @saintpeter to jump in on this since they are more familiar with the issue.

[SaintPeter]

The issue we're seeing occurs when we run the gatsby build process. I've traced it down to the static-generation file where the webpack.run occurs. The webpack appears to fail silenetly - no production bundle is created.

Here is the output with DEBUG=* set:

$ gatsby build
Generating static html pages
  gatsby:glob globbed +0ms 601 pages
  gatsby:static generating static site +0ms
  gatsby:webpack-config Loading webpack config for stage "static" +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Parse start +90ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Parse stop +20ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Start set AST +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: End set AST +20ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Start module formatter init +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: End module formatter init +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Start transformer builtin-pre +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Finish transformer builtin-pre +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Start transformer builtin-basic +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Finish transformer builtin-basic +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Start transformer builtin-advanced +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Finish transformer builtin-advanced +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Start transformer builtin-modules +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Finish transformer builtin-modules +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Start transformer builtin-trailing +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Finish transformer builtin-trailing +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Generation start +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\utils\static-entry.js: Generation end +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Parse start +90ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Parse stop +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Start set AST +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: End set AST +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Start module formatter init +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: End module formatter init +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Start transformer builtin-pre +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Finish transformer builtin-pre +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Start transformer builtin-basic +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Finish transformer builtin-basic +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Start transformer builtin-advanced +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Finish transformer builtin-advanced +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Start transformer builtin-modules +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Finish transformer builtin-modules +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Start transformer builtin-trailing +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Finish transformer builtin-trailing +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Generation start +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\gatsby-helpers.js: Generation end +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Parse start +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Parse stop +20ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Start set AST +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: End set AST +20ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Start module formatter init +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: End module formatter init +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Start transformer builtin-pre +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Finish transformer builtin-pre +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Start transformer builtin-basic +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Finish transformer builtin-basic +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Start transformer builtin-advanced +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Finish transformer builtin-advanced +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Start transformer builtin-modules +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Finish transformer builtin-modules +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Start transformer builtin-trailing +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Finish transformer builtin-trailing +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Generation start +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\create-routes.js: Generation end +30ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Parse start +60ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Parse stop +10ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Start set AST +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: End set AST +10ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Start module formatter init +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: End module formatter init +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Start transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Finish transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Start transformer builtin-basic +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Finish transformer builtin-basic +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Start transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Finish transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Start transformer builtin-modules +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Finish transformer builtin-modules +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Start transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Finish transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Generation start +0ms
  babel [BABEL] c:\mean\wiki-generator\app.js: Generation end +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Parse start +10ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Parse stop +10ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Start set AST +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: End set AST +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Start module formatter init +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: End module formatter init +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Start transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Finish transformer builtin-pre +10ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Start transformer builtin-basic +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Finish transformer builtin-basic +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Start transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Finish transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Start transformer builtin-modules +10ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Finish transformer builtin-modules +10ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Start transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Finish transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Generation start +0ms
  babel [BABEL] c:\mean\wiki-generator\html.js: Generation end +10ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Parse start +60ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Parse stop +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Start set AST +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: End set AST +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Start module formatter init +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: End module formatter init +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Start transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Finish transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Start transformer builtin-basic +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Finish transformer builtin-basic +10ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Start transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Finish transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Start transformer builtin-modules +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Finish transformer builtin-modules +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Start transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Finish transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Generation start +0ms
  babel [BABEL] c:\mean\wiki-generator\wrappers\md.jsx: Generation end +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Parse start +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Parse stop +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Start set AST +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: End set AST +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Start module formatter init +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: End module formatter init +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Start transformer builtin-pre +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Finish transformer builtin-pre +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Start transformer builtin-basic +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Finish transformer builtin-basic +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Start transformer builtin-advanced +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Finish transformer builtin-advanced +10ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Start transformer builtin-modules +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Finish transformer builtin-modules +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Start transformer builtin-trailing +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Finish transformer builtin-trailing +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Generation start +0ms
  babel [BABEL] c:\Users\rex_schrader\AppData\Roaming\npm\node_modules\gatsby\dist\isomorphic\wrappers\html.js: Generation end +0ms
  gatsby:glob globbed +3s 601 pages
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Parse start +2s
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Parse stop +60ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Start set AST +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: End set AST +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Start module formatter init +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: End module formatter init +10ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Start transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Finish transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Start transformer builtin-basic +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Finish transformer builtin-basic +10ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Start transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Finish transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Start transformer builtin-modules +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Finish transformer builtin-modules +50ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Start transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Finish transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Generation start +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\_template.jsx: Generation end +20ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Parse start +10ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Parse stop +40ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Start set AST +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: End set AST +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Start module formatter init +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: End module formatter init +10ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Start transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Finish transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Start transformer builtin-basic +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Finish transformer builtin-basic +30ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Start transformer builtin-advanced +10ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Finish transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Start transformer builtin-modules +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Finish transformer builtin-modules +10ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Start transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Finish transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Generation start +10ms
  babel [BABEL] c:\mean\wiki-generator\pages\en\_template.jsx: Generation end +20ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Parse start +30ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Parse stop +20ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Start set AST +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: End set AST +10ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Start module formatter init +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: End module formatter init +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Start transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Finish transformer builtin-pre +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Start transformer builtin-basic +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Finish transformer builtin-basic +10ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Start transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Finish transformer builtin-advanced +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Start transformer builtin-modules +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Finish transformer builtin-modules +10ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Start transformer builtin-trailing +0ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Finish transformer builtin-trailing +23ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Generation start +1ms
  babel [BABEL] c:\mean\wiki-generator\pages\es\_template.jsx: Generation end +3ms

It returns to my shell prompt after this. Note that while this output is from a windows machine, I have seen identical behavior on a linux (Ubuntu 15.10) and OSX boxes as well.

[justincorrigible]

subscribing :+1:

[KyleAMathews]

Quick questions — are any html pages being generated? If there are, are all?

Also the Gatsby build steps are 1) build the static html pages and then 2) build the bundle.js/styles.css. It looks like your build is failing before it gets to building the bundle.js

[KyleAMathews]

Also to investigate why the error isn't being displayed.

[SaintPeter]

No, no html pages are being generated at all, neither is bundle.js. If I delete down to about 150 input .md files then the static HTML files ARE generated.

NOTE: I can run the site in develop mode and it appears to work properly (with some routing related weirdness which may or may not be related). All of the HTML files are pulled into the bundle.js file as embedded modules.

The webpack build time with ~600 files is about 25 seconds.

[KyleAMathews]

This is odd... perhaps this is a memory issue? I haven't looked at how much memory Gatsby uses — perhaps check that?

[KyleAMathews]

Also with the full ~600 files, does it always quit at the same point?

[SaintPeter]

The thing is that with the prior revision of Gatsby (before the changes to enable react-router 2.0), we were able to generate the site with no problem.

I'll see if I can find a way to determine if there are memory issues. I guess I would not expect webpack (or node) to fail silently if there was an out of memory error. Also, the compiled bundle.js with all of the files is only about 3.5MB (for develop mode), and the input data is somewhat less than that. Node has a default memory size of 500MB on 32bit and 1.5GB for 64bit (which I'm running). Unless something exponential is happening in memory, I don't see how we'd be anywhere close to that.

Yes, the process always quits at the same point - at least from the perspective of the debug output.

I added debug statements to a local fork of Gatsby at the webpack.run() callback function and they were never reached.

[KyleAMathews]

Yeah, an OOM error doesn't seem that likely. Yeah, it seems the thing to do now is to narrow down exactly where the failure is happening.

[KyleAMathews]

I just created a test site with ~900 pages and it built just fine. This suggests it's not a size problem but there's something in your code that's throwing an error. It is absolutely a problem that the error is being swallowed somewhere.

[KyleAMathews]

Also note on performance. Running gatsby develop took ~25 seconds to start and gatsby build took 53 seconds.

[scottnonnenberg]

Watch the memory usage of the process as you're trying to generate for many, many input files. Node will just fall over and die if memory usage gets too high: http://stackoverflow.com/questions/7193959/memory-limit-in-node-js-and-chrome-v8

More generally, I think we may need an option to turn off bundle.js generation, because, for sure, you don't want a bundle.js that includes 10k articles. Back of the envelope math gets us to multiple megabytes very quickly. My 34-post blog has a 218kb gzipped bundle.js.

[KyleAMathews]

@scottnonnenberg yeah, for a very large site you'll want to split the bundle.js into smaller chunks https://github.com/gatsbyjs/gatsby/issues/10

The ability to just turn it off would be a decent idea as well for simple blogs as most incoming traffic just hit one page and then leave so loading other pages wouldn't help much. Per-page bundles could be useful as well. Perhaps you could load the js bundle for the page you're on as well as always loading the home page bundle since that's what someone is most likely to visit from a landing page.

[BerkeleyTrue]

@KyleAMathews After some debuging we've gotten pretty close to where things fail and it seems to fail deep in webpack. It gets to this point, https://github.com/webpack/webpack/blob/72e8dd01475841b5123607efd8b3c7b892c1a32a/lib/Compiler.js#L255, but the callback is never called.

[KyleAMathews]

So weird!

DEEP IN THE BOWELS OF WEBPACK --> this would be a terrifying + dark scene for a frontend dev movie :-D

Perhaps try a debugger from that point? https://nodejs.org/api/debugger.html

[BerkeleyTrue]

Alright, following the rabbit hole even further has ended here https://github.com/markdalgleish/static-site-generator-webpack-plugin/blob/master/index.js#L69. The callback passed into the nodeify method is never called.

[bartekus]

I guess I would not expect webpack (or node) to fail silently if there was an out of memory error. I wouldn't, in-fact out of memory failures can be most strange to diagnose since in my experience they can break promise/callback chains, rendering them useless and causing programs to run into the ground while displaying unrelated and misleading issues/problems. For example memory-leak can cause [Error] Error: Could not locate the bindings file. Tried: as it will prevent proper execution of node-gyp rebuild which a lot of modules require to operate. This will not be immediately apparent however. I would try to compartmentalize the code so that the execution is divided into more manageable segments. I'm not convinced that 10k worth of documents pages should compiled in a single pass.

[KyleAMathews]

@BerkeleyTrue ok I'd try next adding a console.log(outputPathName, render) to https://github.com/markdalgleish/static-site-generator-webpack-plugin/blob/master/index.js#L58

and then console.log(err) to https://github.com/markdalgleish/static-site-generator-webpack-plugin/blob/master/index.js#L65

Perhaps one of the render functions is causing troubles.

[kuetemeier]

I also tried Gatsby with 5000 markdown articles to compare it to Hugo 5000 Posts in 7 seconds, and ended up with the same out of memory error.

If you wanna try it out, I have just created a repo that show's the error: https://github.com/jkuetemeier/gatsby-5000

[KyleAMathews]

@jkuetemeier nice test! I'll play around with this later. You can raise the memory limit for Node pretty easily but I'd need to do some research on how to enable that in a command-line app like Gatsby. If this interests you Jörg this would be a great PR. Expose something like --memory 4096.

Also to be clear, it doesn't seem like the original issue has anything to do with memory as otherwise they'd get they'd see the error you're seeing. But to support 10,000 post sites, we'll definitely need the ability to raise the memory limit.

Also for those interested in this issue, check out https://github.com/gatsbyjs/gatsby/issues/151

[BerkeleyTrue]

@KyleAMathews

ok I'd try next adding a console.log(outputPathName, render)

I tried it. It seems like the individual promise have no isssue, but the Promise.all seems to swallow some error. Does this indicate some issue with out temple/markdown files?

You can raise the memory limit for Node pretty easily

Have you considered using git to reduce the number of items Gatsby needs to build. Gatsby could check the git diff and rebuild just those files that have changed. This would reduce make memory only an issue on template changes.

[KyleAMathews]

@BerkeleyTrue incremental production builds would be very cool. The problem though isn't rebuilding the underlying content but bundle.js. This is something that'll probably need done in Webpack https://www.google.com/search?q=webpack+incremental+production+builds&oq=webpack+incremental+productio&aqs=chrome.1.69i57j69i59.4450j0j7&sourceid=chrome&ie=UTF-8

Did you add the console.err in the catch? That's where you'll see if there's an error that's thrown in an individual promise. Also perhaps put a console.log in the then callback.

[KyleAMathews]

@BerkeleyTrue added an issue for incremental production builds @ #179

[BerkeleyTrue]

Did you add the console.err in the catch? That's where you'll see if there's an error that's thrown in an individual promise. Also perhaps put a console.log in the then callback.

Yup, it was a no go. The then gets called from the individual promises, no error is thrown, but if I put a then after the promise all, it never gets called.

[kuetemeier]

Incremental builds may be very cool, but sometime we'll need to rebuild the whole site at once... e.g. when we host our apps on Heroku etc.

[SaintPeter]

FYI, RE: Memory usage - I watched the process with htop and it maxed out at around 760MB, which is about half the default max (1.5GB) for Node at x64 (which I confirmed I am running).

[KyleAMathews]

@BerkeleyTrue @SaintPeter this sounds like it might be a bug in the webpack static site generator then. If you post an issue there, tag me on it so I can follow along.

[KyleAMathews]

@jkuetemeier absolutely. Incremental builds would be an optimization not a requirement. @jquense linked to this interesting issue in the Webpack repo about caching intermediate states in Webpack which would solve the problem nicely. https://github.com/gatsbyjs/gatsby/issues/179#issuecomment-194496031

[KyleAMathews]

@jkuetemeier I got your 5000 page site to build after a bit of work.

I had to first increase the memory limit in bin/gatsby.js and bin/gatsby-build.js by changing the first line to #!/usr/bin/env node --max_old_space_size=16192

That solved the OOM problem. But I then ran into this error: Error: ENFILE: file table overflow which I solved by upping the ulimit and maxfiles.

The build took 8:18 and turned out a bundle.js of 12m or 2.2mb after gzipping. So 5000+ pages is definitely doable but there's lots of optimizations that could be done.

[KyleAMathews]

Much of the build time I'll note is spent in building the bundle.js. Generating the HTML pages is pretty quick.

[kuetemeier]

Great work.

[SaintPeter]

@KyleAMathews Why is so much time spent building the bundle.js? Shouldn't the vast majority of the code on the 5k articles be identical? Would a better template structure reduce duplicate code?

[justincorrigible]

While I doubt this would help, but I've received these warnings the other day...

Could it have anything to do with the issue?

[KyleAMathews]

@SaintPeter Gatsby pulls all the metadata for each page into the bundle.js as well so the frontmatter + html of the body of the page — this is why clicking around a Gatsby site is so fast because all information necessary to render other pages is already loaded into the browser.

But in any case, building the bundle.js for production is mostly slow because of the minification step w/ uglify.js.

[KyleAMathews]

@hallaathrad was this from the browser console?

[justincorrigible]

@KyleAMathews yes. using FirefoxDeveloperEditon 46.0a2 @ OSX 1011.4 (15E56a). Doesn't happen on any other site or browser.

[KyleAMathews]

@hallaathrad thanks. Yeah probably doesn't then have something to do with this issue.

[KyleAMathews]

@BerkeleyTrue @SaintPeter what's your status?

[KyleAMathews]

Just ran into this problem. React-Router 2 gets very slow as you add more child routes https://github.com/reactjs/react-router/issues/3215 This is causing problems even for my little ~98 post blog.

[BerkeleyTrue]

I'm afraid I haven't had the bandwidth to investigate further. @SaintPeter Where you able to make progress with the bundle creation disabled?

[BerkeleyTrue]

@KyleAMathews Looks like the issue stemmed from an error in our template file. Looks like something unrelated was swallowing the error, though.

I was able to finally get an error printout by delete large chunks of files and trying a build. Looks like it was unrelated to the size of the our project, so I think this issue can be closed.

[KyleAMathews]

So what was swallowing the error was in your app?

Ok glad you found it!

[parkerproject]

@KyleAMathews I don't think you addressed the original question which was if Gatsby is designed to handle a large number of pages. I know @QuincyLarson mentioned 10,000 articles, but what of a million pages? that is possible with a large online magazine. Let me know your thoughts

[KyleAMathews]

That's the direction we're headed. It doesn't yet support a million page site but it will.

[parkerproject]

Awesome, thanks for clarifying that.

[JoseFMP]

@KyleAMathews do you know have more information on this? I.e. you wrote It doesn't yet support a million page site but it will. and we are very interested in this feature. Are there any timelines? For us currently Gatsby cannot build more than ... 10% of the amount of pages we want to build. Shouldn't there be an option to let Gatsby push the content incrementally to the file system and release memory? Just wondering.

[kuetemeier]

If you have such high amount of pages, perhaps Hugo may be the right tool. We have moved our site completely to Hugo.

[mokimo]

As far as I've read in an AMA incremental builds are in the planning and thus large online magazines wouldn't have to rebuild a million articles anymore