Open ardavank opened 6 months ago
A 30k blog post? π€― why do you have so many?
Yes, it seems like we should not use JSON.stringify
but instead use a serialization lib supporting streaming.
For example https://github.com/dominictarr/JSONStream#jsonstreamstringifyopen-sep-close
Note to myself: I'll probably take this opportunity to encapsulate this streaming, and expose simpler, more testable interfaces to create route data bundles because I'm not a fan of our historic plugin actions API for that:
// Create a blog archive route
const archiveProp = await createData(
`${docuHash(archiveUrl)}.json`,
JSON.stringify({blogPosts: listedBlogPosts}, null, 2),
);
addRoute({
path: archiveUrl,
component: blogArchiveComponent,
exact: true,
modules: {
archive: aliasedSource(archiveProp),
},
});
That would be more convenient to have everything handled for you, and just write:
addRoute({
path: archiveUrl,
component: blogArchiveComponent,
exact: true,
props: {
archive: {blogPosts: listedBlogPosts},
},
});
Note we'd still need to keep a way to create data bundles independent from routes (+ streaming support), because those data bundles can be shared between routes, reducing the amount of data to load when navigating from one route to another. (although this data could probably be added as a routeContext
)
@slorber thanks for looking into this issue, I'm replacing Wordpress with Docusaurus and I'm migrating an existing news website. The reason for this is the simplicity, speed and the low maintenance cost of Docusaurus!
You can see it in action: https://mesghalapp.com/en/news
Currently I have two blockers:
1- Is the issue with the JSON.stringify
2- The 26mb file limit of Cloudflare
The screenshot above is for including only ~2K blogs
The JSON.stringify({blogPosts: listedBlogPosts}
creates a massive file especially when the blogs are written in other languages that need to be encoded as well.
is possible to split this file into smaller files as well?
https://mesghalapp.com/en/news/archive/
To be honest I don't think Docusaurus is designed to support that kind of scale. It seems you have 2k entries just for 2024, and if it keeps growing at the same pace the build time will quickly become unsustainable.
You'd rather use a docs framework that supports server-side-rendering.
Note that the blog archive page can be disabled with option archiveBasePath: null
, so it might unlock you temporary:
https://docusaurus.io/docs/api/plugins/@docusaurus/plugin-content-blog#archiveBasePath
But even with that solution, I doubt Docusaurus will be the best choice for your need.
@slorber thanks for info, currently I'm only keeping the news for 3 months by deleting the older ones to manage this limitation. I feel like the build process can be improved, can you please guide me on how to debug the build and get more performance log for each step of the progress? I'm interested in improving the build performance
I'm interested in improving the build performance
You are lucky because I'm actively working on Docusaurus performance issues right now.
The upcoming v3.2 release will be faster and have some basic perf logging that you can turn on with DOCUSAURUS_PERF_LOGGER=true
(internal usage, not publicly documented for now). You can try using the latest canary to benefit from these improvements. Afaik canary doesn't have yet any breaking change from v3 so it should be fine.
However, it does not fix all the problems yet, and the main unfixed bottleneck remains bundling your app with Webpack for both client consumption and SSR. Also, the bundle we assemble for server/node usage is historically a huge single JS file, that causes memory issues during the SSG phase.
Yes the build process can be improved, but this is likely quite technical and I'd prefer it to handle it myself. Most likely we will try to swap Webpack by Rspack soon and provide a flag to enable Rspack to provider an incremental migration path. But Rspack is not yet 100% retrocompatible with Webpack so it might not even work right now.
@slorber
I tried version 0.0.0-5861
, and here are the results:
> mesghal-app-com@0.0.0 build
> docusaurus build --locale en --out-dir build/en
[PERF] Get locales to build: 0.197ms
[INFO] [en] Creating an optimized production build...
[PERF] Load - loadContext: 149.412ms
[PERF] Plugins - initPlugins: 123.724ms
[PERF] Plugin - loadContent - docusaurus-plugin-sitemap@default: 0.609ms
[PERF] Plugin - loadContent - docusaurus-plugin-google-gtag@default: 0.646ms
[PERF] Plugin - loadContent - docusaurus-bootstrap-plugin@default: 0.703ms
[PERF] Plugin - loadContent - docusaurus-mdx-fallback-plugin@default: 0.865ms
[PERF] Plugin - loadContent - docusaurus-theme-classic@default: 22.973ms
[PERF] Plugin - loadContent - docusaurus-plugin-content-pages@default: 110.539ms
[PERF] Plugin - loadContent - docusaurus-plugin-content-blog@default: 1.233s
[PERF] Plugins - loadContent: 1.234s
[PERF] Plugins - contentLoaded - docusaurus-plugin-sitemap@default: 0.081ms
[PERF] Plugins - contentLoaded - docusaurus-theme-classic@default: 0.13ms
[PERF] Plugins - contentLoaded - docusaurus-bootstrap-plugin@default: 0.207ms
[PERF] Plugins - contentLoaded - docusaurus-mdx-fallback-plugin@default: 0.254ms
[PERF] Plugins - contentLoaded - docusaurus-plugin-google-gtag@default: 1.55ms
[PERF] Plugins - contentLoaded - docusaurus-plugin-content-pages@default: 3.212ms
[PERF] Plugins - contentLoaded - docusaurus-plugin-content-blog@default: 803.121ms
[PERF] Plugins - contentLoaded: 814.188ms
[PERF] Plugins - allContentLoaded - docusaurus-plugin-content-blog@default: 0.085ms
[PERF] Plugins - allContentLoaded - docusaurus-plugin-content-pages@default: 0.089ms
[PERF] Plugins - allContentLoaded - docusaurus-plugin-sitemap@default: 0.115ms
[PERF] Plugins - allContentLoaded - docusaurus-theme-classic@default: 0.134ms
[PERF] Plugins - allContentLoaded - docusaurus-plugin-google-gtag@default: 0.152ms
[PERF] Plugins - allContentLoaded - docusaurus-bootstrap-plugin@default: 0.168ms
[PERF] Plugins - allContentLoaded - docusaurus-mdx-fallback-plugin@default: 0.185ms
[PERF] Plugins - allContentLoaded: 1.043ms
[PERF] Plugins - loadPlugins: 2.176s
[PERF] Load - loadPlugins: 2.176s
[PERF] Load - loadSiteMetadata: 0.478ms
[PERF] Load - loadCodeTranslations: 1.057ms
[PERF] Load - createSiteFiles: 195.6ms
[PERF] Loading site: 2.529s
[PERF] Creating webpack configs: 334.828ms
[PERF] Deleting previous client manifest: 0.569ms
β Client
Compiled successfully in 5.21m
β Server
β Client βββββββββββββββββββββββββ cache (99%) shutdown IdleFileCachePlugin
stored
β Server
[PERF] Bundling: 5:21.375 (m:ss.mmm)
[PERF] Reading client manifest: 17.088ms
[PERF] Compiling SSR template: 1.507ms
SSG - Load server bundle
[PERF] SSG - Load server bundle: 44.964ms
[PERF] SSG - Server bundle size = 29.351 MB
[PERF] SSG - Evaluate server bundle: 842.003ms
[PERF] Loading App renderer: 887.578ms
[PERF] Generate static files: 1:55.173 (m:ss.mmm)
[PERF] Executing static site generation: 1:56.080 (m:ss.mmm)
[PERF] Deleting server bundle: 2.146ms
[PERF] Executing postBuild(): 215.985ms
[PERF] Executing broken links checker: 809.84ms
[SUCCESS] Generated static files in "build/en".
[INFO] Use `npm run serve` command to test your build locally.
it looks like that Generate static files
and Executing static site generation
steps are taking ~4 minutes.
Is this expected?
It's more 2min than 4min, because the log is unclear but one ask is composed of another.
And yes that seems expected that rendering, minifying and wriing thousands of static files takes time.
For a blog, the number of pages to generate can grow quickly depending on your usage of tags and your pagination setting.
What takes the most time remains the bundling phase, which has not been optimized yet.
I'm currently in the process of migrating from 2.4.3 to 3.4.0 and encountered the RangeError: Invalid string length
error, although I'm not sure it's related:
Full error portion of stack trace:
Error: HookWebpackError: Invalid string length
at makeWebpackError (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/HookWebpackError.js:48:9)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:3181:12
at eval (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:24:1)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
-- inner error --
RangeError: Invalid string length
at ConcatSource.source (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/node_modules/webpack-sources/lib/ConcatSource.js:76:20)
at CachedSource.source (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/node_modules/webpack-sources/lib/CachedSource.js:101:48)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/optimize/RealContentHashPlugin.js:170:59
at fn (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:544:19)
at Hook.eval [as callAsync] (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:14:17)
at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/Hook.js:18:14)
at cont (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:3178:34)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:3228:10
at symbolIterator (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:3485:9)
at done (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:3527:9)
at /home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2830:7
at done (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2865:11)
at /home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2818:7
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:4841:18
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/HookWebpackError.js:68:3
at _done (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
at Hook.eval [as callAsync] (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:39:22)
at Cache.store (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:113:20)
at ItemCacheFacade.store (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/CacheFacade.js:141:15)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:4838:25
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:91:6
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:44:11
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/cache/IdleFileCachePlugin.js:78:9
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:94:6
at eval (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:30:1)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Error: caused by plugins in Compilation.hooks.processAssets
RangeError: Invalid string length
at ConcatSource.source (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/node_modules/webpack-sources/lib/ConcatSource.js:76:20)
at CachedSource.source (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/node_modules/webpack-sources/lib/CachedSource.js:101:48)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/optimize/RealContentHashPlugin.js:170:59
at fn (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:544:19)
at Hook.eval [as callAsync] (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:14:17)
at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/Hook.js:18:14)
at cont (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:3178:34)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:3228:10
at symbolIterator (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:3485:9)
at done (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:3527:9)
at /home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2830:7
at done (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2865:11)
at /home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2818:7
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:4841:18
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/HookWebpackError.js:68:3
at _done (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
at Hook.eval [as callAsync] (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:39:22)
at Cache.store (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:113:20)
at ItemCacheFacade.store (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/CacheFacade.js:141:15)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:4838:25
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:91:6
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:44:11
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/cache/IdleFileCachePlugin.js:78:9
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:94:6
at eval (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:30:1)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Error: Error: Unable to build website for locale en.
at tryToBuildLocale (/home/runner/work/pan.dev/pan.dev/node_modules/@docusaurus/core/lib/commands/build.js:54:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /home/runner/work/pan.dev/pan.dev/node_modules/@docusaurus/core/lib/commands/build.js:65:9
at async mapAsyncSequential (/home/runner/work/pan.dev/pan.dev/node_modules/@docusaurus/utils/lib/jsUtils.js:20:24)
at async Command.build (/home/runner/work/pan.dev/pan.dev/node_modules/@docusaurus/core/lib/commands/build.js:63:5) {
[cause]: HookWebpackError: Invalid string length
at makeWebpackError (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/HookWebpackError.js:48:9)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:3181:12
at eval (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:24:1)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
-- inner error --
RangeError: Invalid string length
at ConcatSource.source (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/node_modules/webpack-sources/lib/ConcatSource.js:76:20)
at CachedSource.source (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/node_modules/webpack-sources/lib/CachedSource.js:101:48)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/optimize/RealContentHashPlugin.js:170:59
at fn (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:544:19)
at Hook.eval [as callAsync] (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:14:17)
at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/Hook.js:18:14)
at cont (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:3178:34)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:3228:10
at symbolIterator (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:3485:9)
at done (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:3527:9)
at /home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2830:7
[INFO] Docusaurus version: 3.4.0
Node version: v20.15.0
at done (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2865:11)
at /home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2818:7
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:4841:18
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/HookWebpackError.js:68:3
at _done (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
at Hook.eval [as callAsync] (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:39:22)
at Cache.store (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:113:20)
at ItemCacheFacade.store (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/CacheFacade.js:141:15)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:4838:25
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:91:6
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:44:11
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/cache/IdleFileCachePlugin.js:78:9
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:94:6
at eval (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:30:1)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
caused by plugins in Compilation.hooks.processAssets
RangeError: Invalid string length
at ConcatSource.source (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/node_modules/webpack-sources/lib/ConcatSource.js:76:20)
at CachedSource.source (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/node_modules/webpack-sources/lib/CachedSource.js:101:48)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/optimize/RealContentHashPlugin.js:170:59
at fn (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:544:19)
at Hook.eval [as callAsync] (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:14:17)
at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/Hook.js:18:14)
at cont (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:3178:34)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:3228:10
at symbolIterator (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:3485:9)
at done (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:3527:9)
at /home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2830:7
at done (/home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2865:11)
at /home/runner/work/pan.dev/pan.dev/node_modules/neo-async/async.js:2818:7
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:4841:18
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/HookWebpackError.js:68:3
at _done (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
at Hook.eval [as callAsync] (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:39:22)
at Cache.store (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:113:20)
at ItemCacheFacade.store (/home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/CacheFacade.js:141:15)
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Compilation.js:4838:25
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:91:6
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:44:11
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/cache/IdleFileCachePlugin.js:78:9
at /home/runner/work/pan.dev/pan.dev/node_modules/webpack/lib/Cache.js:94:6
at eval (eval at create (/home/runner/work/pan.dev/pan.dev/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:30:1)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
}
error Command failed with exit code 1.
The site is rather large but currently builds fine on 2.4.3, although it takes upwards of 45 minutes. Should 3.x have necessarily impacted Docusaurus' ability to scale to large sites? Any other troubleshooting steps or workarounds I can/should try? I'm planning to scour the docs, release notes and issues to see if I may have missed someone else posting a solution. Thanks!
@sserrata unfortunately I don't really know what's causing the issue above, but it looks like your JS assets are too big.
Maybe under v2, you were just under the limit, and we added extra React components and features in v3 that contribute to the bundle size increase that makes you reach that limit?
I don't know for sure but I don't remember anything that could have impacted the scalability.
What I see is that the error gets emitted in node_modules/webpack/lib/optimize/RealContentHashPlugin.js:170:59
which looks responsible for computing hashes from content string. Even though the Webpack error message is unclear, you can probably add a breakpoint or try/catch/log here to figure out which file exactly is causing the problem:
Have you read the Contributing Guidelines on issues?
Prerequisites
npm run clear
oryarn clear
command.rm -rf node_modules yarn.lock package-lock.json
and re-installing packages.Description
docusaurus build returns
RangeError: Invalid string length
when building the website with a very large number of blogs (~30K)Error:
Looking at the code, it looks like
JSON.stringify({ blogPosts: listedBlogPosts }
is the problematic partReproducible demo
No response
Steps to reproduce
Expected behavior
Actual behavior
it throws
RangeError: Invalid string length
Your environment
Self-service