Extremely slow compilation and high resource usage, even on v4 with turbopack

[oprypkhantc]

Describe the bug

Hey.

On the latest v4 release (version .22), the nextra-docs template takes >60sec to render the "Documentation" page, while using 300-400% CPU and 1.8GB of RAM on a modern MacBook Pro.

In of itself, a cold start taking some time isn't a huge problem; but even next renders (after changing a single page of content) still take >10sec to complete, all while using 2GB of RAM. Turbopack doesn't help at all here; it just uses 8x-10x times RAM, without any visible improvement to render time.

There is an old closed issue, mentioning that NextJS development is slow, but this really isn't the case with an example app from NextJS - where builds take <100ms. The difference is more than 100x in this case between a clean example NextJS app and a Nextra Docs template, which seems suspicious :)

To Reproduce

1. Clone the examples/docs template from v4-v2 branch
2. Install deps, run the NextJS server: yarn run next dev
3. Navigate to http://localhost:3000/docs, wait for it to render
4. Change anything in content/index.mdx
5. See the change after 10-20sec

Expected behavior

Rerenders should probably be faster.

Screenshots

An example of how much resources initial render with Turbopack uses (some of it is shared memory, but still):

[dimaMachina]

Changes from content folder may be slow compared to page.{jsx,tsx,mdx} convention because it uses dynamic import

https://github.com/shuding/nextra/blob/bf029ce8c2208a2c37c55dbbd67f35b56bc3ce98/packages/nextra/src/client/pages.ts#L11-L13

And it seems like Webpack/Turbopack precompiles all files from it.

Did you notice the same problems on migrated nextra.site website (docs folder)?

See the change after 10-20sec

On my mac I don't need to wait this time, everything happens less for a second

P.S. Also, solving this issue should improve performance for Turbopack projects

[oprypkhantc]

Did you notice the same problems on migrated nextra.site website (docs folder)?

No, in fact it's as fast as you describe it - hot reload updates take less than a second, and the compilation itself is ~100ms. It's as quick as I expect it to be :)

So having no previous experience with NextJS, as I understand it, the docs example has a fallback catch-all route that causes all of the assets to be recompiled, and the migrated Nextra site just uses Next's static routing? And the solution is to ditch the dynamic import and dynamic routing?

[dimaMachina]

and the migrated Nextra site just uses Next's static routing?

Nextra 4 has 2 modes

• Next.js App Router page.{jsx,tsx} convention but also with page.{md,mdx} extensions
• content dir, you may think is as Next.js Pages Router which rendered via single catch-all [[...mdxPath]].{jsx,tsx} route

[!TIP] They can work separately and together.

---

the docs example has a fallback catch-all route that causes all of the assets to be recompiled

I can't give the right response to this question...

When I add console logs in Nextra's Webpack loader, and make some changes in MDX files - I see logs with the file path of the changed file (only the initial loading of the project contains logs from all files because Nextra builds pageMap - array of existing files/folder and their exported front matter or metadata, to show proper sidebar and navigation)

[!NOTE]

content dir was made an alternative to Next.js Pages Router to make smooth migration, it's still experimental so it may have some performance issues, maybe some folks in the future will help with the improvement of this mode

[oprypkhantc]

Nextra 4 has 2 modes

Thank you, that cleared it up for me.

When I add console logs in Nextra's Webpack loader, and make some changes in MDX files - I see logs with the file path of the changed file (only the initial loading of the project contains logs from all files because Nextra builds pageMap - array of existing files/folder and their exported front matter or metadata, to show proper sidebar and navigation)

I'll try to debug it later. However, there's one interesting finding: when I start the example docs app from the cloned Nextra repository, with package.json referencing nextra* dependencies from workspace, it flies. It's fast. Just as fast as the website app.

However, if I extract the demo into a separate folder (not a subpath of the Nextra repository), then replace those workspace versions with 4.0.0-app-router.22 and 4.0.0-app-router.22, and install dependencies with yarn - it suddenly is slow again.

Which makes me think the problem is either in the commits after version 4.0.0-app-router.22, or in the dev/prod build 🤔

[dimaMachina]

Interesting, thank for investigation, will be appreciated if you could debug or provide reproduction, so I could debug too when I'll have time 🙏

[oprypkhantc]

Ha. I found the culprit. It's the repository.getFileLatestModifiedDateAsync. No wonder you couldn't reproduce it - our repo has >250k commits, so simply getting a modification date (for an uncommitted file) takes ~12sec. Commenting out the git stuff fixes the problem completely, and so does committing a file.

Looking through the simple-git's code, it looks like as long as the file that's being checked was committed recently, it will be fast, but I'm sure that it will become slow again once it has tens of thousands of commits to go through before finding the file.

Unfortunately I do not see a way to disable these checks. I can see how it can be useful, but not at the expense of performance 😄 I'm not much of a JS dev, but I can PR an option to disable the Git/last edit stuff - something like a timestamps: false option in the config?

[dimaMachina]

Amazing @oprypkhantc, I can reproduce it too!

I think the simplest fix will be to cache the result of repository.getFileLatestModifiedDateAsync

[oprypkhantc]

Even with caching, the initial build would still take minutes instead of seconds, due to having to look up each individual file 🤔 Maybe only calculate timestamps for production builds?

[dimaMachina]

yeah, changed, try https://github.com/shuding/nextra/releases/tag/nextra%404.0.0-app-router.24

[dimaMachina]

@oprypkhantc Even though we haven't calculated the last edit time for dev, I still feel we need some static value because we have this field in _meta file to hide this timestamp https://nextra-v2-164w6708l-shud.vercel.app/docs/docs-theme/built-ins/layout#toggle-visibility-for-timestamp

So it will be better for dx to see something on dev, something like Date.now()

[oprypkhantc]

Indeed working as expected with .24 :) Thank you!

[oprypkhantc]

So it will be better for dx to see something on dev, something like Date.now()

Yes, makes sense. Dev builds will be closer to production, which is always good

[oprypkhantc]

By the way, @dimaMachina, are you also seeing that high of a resource usage? At peaks running the dev server on the docs example uses >2GB, and even after peaks it stays at ~1GB while doing nothing.

I know that Webpack is very inefficient, and I even enabled Next's webpackMemoryOptimizations setting, but it still seems a bit high of a usage. Also, since Turbopack is written with Rust, it should be super efficient, but in practise it just makes things worse by using ~2.5GB while again doing nothing at all.

Also I understand this may not be Nextra's issue at all, but it's hard to tell with these kind of issues. Especially given that an example NextJS app (with TypeScript and Tailwind) only uses ~600MB with Turbopack. Anyway, I first wanted to confirm that you're seeing this too, and that the issue you mentioned (https://github.com/vercel/next.js/issues/71959) is not related. Then I'll report this to NextJS and see what they say :)

[dimaMachina]

Hm.. I don't know how I can help with this issue, can you try to test:

• resource usage of Nextra website with only 1 page
• comparison of resource usage with @next/mdx for example

[oprypkhantc]

Nextra with just the blog page is still the same - 2GB peak / 1GB after without Turbopack, and 2.5GB with Turbopack. NextJS with @next/mdx uses 800MB with Turbopack, and all of my CPUs :(

The CPU issue then is a 100% Turbopack's fault then; memory usage - can it be caused by additional dependencies of Nextra? I did include Tailwind in the example NextJS app, but other than that it's the clean example app

[dimaMachina]

Try to disable one by one remark/rehype plugins to see which causes an issue

[oprypkhantc]

Even removing literally all plugins (except for recma ones) and even moving to .md, memory usage still peaked at 2.25GB and stays at 1.9GB.

Some plugins I didn't remove because they set some properties on the AST - I just made sure those set static values and do nothing else.

[dimaMachina]

Thank you 🙏 no more ideas for now

[dimaMachina]

maybe it's due this

https://github.com/shuding/nextra/blob/c095f98c8bf09bba6c135cd3d8cf48debfb08570/packages/nextra/src/server/loader.ts#L124-L127

we need to call only 1 time outside of loader