Turbopack dev server and HMR consuming too much memory and freezing at code changes.

devhik0 commented 4 months ago

Link to the code that reproduces this issue

To Reproduce

Run next dev --turbo and check Memory values at task manager, htop etc. Also check VS Code's process explorer via F1 > Open Process Explorer and look for "<path>\nodejs\node.exe" <path-to-app>\node_modules\next\dist\server\lib\start-server.js

Current vs. Expected behavior

That process above is consuming around 1GB memory at idle state (just running and no code change), thus code change reflections taking around > 30s to appear on UI. Here is the latest hmr result:

○ Compiling /tasks ...
✓ Compiled in 39.9s (sometimes around 60)
✓ Compiled /tasks in 18.2s

Expected: faster rebuilds to see what happens at UI in short time and less memory consumption

I think there is a memory leak at somewhere in process.

Provide environment information

Operating System:
  Platform: win32
  Arch: x64
  Version: Windows 11 Home Single Language
  Available memory (MB): 16280
  Available CPU cores: 4
Binaries:
  Node: 20.10.0
  bun: 1
Relevant Packages:
  next: 14.2.3 // Latest available version is detected (14.2.3).
  eslint-config-next: 14.2.3
  react: 18.3.1
  react-dom: 18.3.1
  typescript: 5.4.5

Which area(s) are affected? (Select all that apply)

Developer Experience, Module Resolution, Performance, Runtime, Turbopack, TypeScript

Which stage(s) are affected? (Select all that apply)

next dev (local)

Additional context

Here i am sharing process info from my system to show what is happening actually:

memory

The process with red is : <path>\nodejs\node.exe" <path-to-app>\node_modules\next\dist\server\lib\start-server.js

Also this is from process explorer at vscode:

process

devhik0 commented 3 months ago

Anything ?

timneutkens commented 3 months ago

Can you try next@canary first, currently it's on the latest stable version but Turbopack is in active development: https://github.com/devhik0/crm-saas/blob/master/client-app/package.json#L35

1GB of memory usage is not much in general, especially if you have large dependencies like heroicons that are quite large (thousands of files)

louisgv commented 3 months ago

Seeing a similar issue running on Mac. It's smoking ~5GB of CPU:

Updated from 14.1.4 -> 14.2.4

devhik0 commented 3 months ago

Well I tried even next 15@rc and without turbo, still my dev server takes around 1gb memory and takes near minutes to compile.

timneutkens commented 3 months ago

In order to help me investigate this I'll ideally need an application that can be run, if you can't provide that (I understand if you can't) please provide the .next/trace and .next/trace.log file.

Please follow these steps which would give me the best picture to investigate:

npm install next@canary (use the package manager you're using) -- We want to make sure you're using the very latest version of Next.js which includes the fixes mentioned earlier.
rm -rf .next
start development using the NEXT_CPU_PROF=1 and NEXT_TURBOPACK_TRACING=1 (regardless of if you're using Turbopack, it only affects when you do use Turbopack) environment variable. E.g.:
- npm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 npm run dev
- yarn: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 yarn dev
- pnpm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 pnpm dev
Wait a few seconds
Open a page that you're working on
Wait till it's fully loaded
Wait a few seconds
Make an edit to a file that holds a component that is on the page
Wait for the edit to apply
Wait a few seconds
Make another edit to the same file
Wait a few seconds
Exit the dev command (ctrl+c)
Upload the CPU traces put in the root of the application directory to https://gist.github.com
Upload the .next/trace file to https://gist.github.com -- Please don't run trace-to-tree yourself, as I use some other tools (e.g. Jaeger) that require the actual trace file.
Upload the .next/trace.log as well, if it's too large for GitHub gists you can upload it to Google Drive or Dropbox and share it through that.
Upload next.config.js (if you have one) to https://gist.github.com
Share it here

devhik0 commented 3 months ago

You know that problematic repo, i will provide env values to run it properly. Here is the project folder and env values, just put lines into a .env.local file inside folder root. project folder

# Supabase
NEXT_PUBLIC_SUPABASE_URL=https://jokkvunyzexyiwypinjd.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6Impva2t2dW55emV4eWl3eXBpbmpkIiwicm9sZSI6ImFub24iLCJpYXQiOjE3MTYzODIyNzEsImV4cCI6MjAzMTk1ODI3MX0.O-_1nkM2Blhm7kjscCL9bC0BIp_yhLeKGVoFKDdo_bw

Clone repo, cd client-app , bun i (to use another package manager, you need to remove bun.lockb before , bun run dev (w/o --turbo flag), click tabs on sidebar to check page loads, try to make changes at related files like app/calendar/page.tsx to see HMR response times. I am seeing it like this:
```
○ Compiling /tasks ...
✓ Compiled in 39.9s (sometimes around 60)
✓ Compiled /tasks in 18.2s
```
Let me know if something goes wrong, thanks for your interest.

timneutkens commented 3 months ago

Can you try to run the steps to grab a trace as well so that we have numbers for your machine? Thanks!

devhik0 commented 3 months ago

Here is files link @timneutkens

devhik0 commented 3 months ago

Anything?

timneutkens commented 2 months ago

Hey @devhik0 sorry I couldn't get around to looking into this yet, could you create a new share link as the old one expired. Thanks!