Too much memory usage on build

alexstine commented 8 months ago

Description

I spun up my new WSL VM this afternoon and discovered the npm run build script is using an insane amount of memory. I received a Node out of memory heap error and had to increase using this environment variable.

export NODE_OPTIONS=--max_old_space_size=6000

It looks like at peak build, it nearly uses 3GB of memory which is absolutely insane. It fails right at the TypeScript build.

Step-by-step reproduction instructions

Spin up a WSL2 Ubuntu 20.04 VM.
Clone the repo.
Run npm i && npm run build.
See error.

Screenshots, screen recording, code snippet

No response

Environment info

N/A.

Please confirm that you have searched existing issues in the repo.

Yes

Please confirm that you have tested with all plugins deactivated except Gutenberg.

Yes

t-hamano commented 8 months ago

Thanks for the report.

I have been running the Gutenberg project on WSL2 for over two years, but I have never encountered such an error, nor do I remember operating max_old_space_size.

It looks like at peak build, it nearly uses 3GB of memory which is absolutely insane.

How can I test to measure this memory?

alexstine commented 8 months ago

@t-hamano If you do the following, you should be able to see memory usage.

Start a screen session with screen -S build.
Run npm run build.
Press Control+A and then D to detach.
Run free -g to output the memory in GB while the build is running. Run the command multiple times.

Part of me wonders if this is only an issue on the initial build now considering cache is likely used on each build after. This might not be reproducible without starting fresh.

Thanks.

t-hamano commented 8 months ago

I performed the following steps.

Cloning the gutenberg repository again.
Run npm i.
Start a screen session with screen -S build.
Run npm run build.
Press Control+A and then D to detach.
Run free -g to output the memory in GB while the build is running. Run the command multiple times.

The average value of the cell where the used column and mem row intersect was 3, and increased to a maximum of 5. Is this measured correctly?

	total	used	free	shared	buff/cache	available
Mem:	31	3	24	0	3	28
Swap:	8	0	8

alexstine commented 8 months ago

Yes, that looks correct. I don't ever remember our TypeScript taking that much memory before. Are you using NVM to manage Node?

t-hamano commented 8 months ago

Are you using NVM to manage Node?

I'm currently using volta.

alexstine commented 7 months ago

I'll leave this open in case anyone else has this problem with NVM, but it seems like I might be the only one. So strange. All in all, the Windows filesystem doesn't perform very well with Node so I guess this should come at no shock to anyone. Mac/Linux handles it much better.

simonhammes commented 6 months ago

I am using NVM on Debian Trixie and just got an OOM while running npm run dev.

This was the last command:

node ./bin/packages/validate-typescript-version.js && tsc --build && node ./bin/packages/check-build-type-declaration-files.js

Running export NODE_OPTIONS=--max_old_space_size=6000 before npm run dev fixes the issue.

alexstine commented 6 months ago

Yes, I am thinking we should either document this or fix the out of memory error.

benridane commented 4 months ago

It appears that the memory-consuming area when tsc --build is run is in packages/components.

212:~/wp1/gutenberg/packages/components$ ../../node_modules/.bin/tsc --build --extendedDiagnostics

<--- Last few GCs --->

[185248:0x62aa000]    45794 ms: Scavenge 972.5 (999.2) -> 969.2 (999.2) MB, pooled: 5 MB, 3.27 / 0.00 ms  (average mu = 0.436, current mu = 0.068) allocation failure;
[185248:0x62aa000]    46331 ms: Mark-Compact 972.5 (999.8) -> 969.1 (1008.5) MB, pooled: 0 MB, 528.37 / 0.01 ms  (average mu = 0.333, current mu = 0.208) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xe18f84 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 2: 0x12102f0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 3: 0x12105c7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0x14400d5  [node]
 5: 0x1459949 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 6: 0x142e018 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 7: 0x142ef45 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0x1407b8e v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
 9: 0x1868ffc v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x1f1e576  [node]
中止 (コアダンプ)

caseymilne commented 3 months ago

I've also been unable to get past the initial build due to memory errors:

> gutenberg@18.8.0 build:packages
> npm run --silent build:package-types && node ./bin/packages/build.js

<--- Last few GCs --->

[8668:00000294C50F0F90]   148165 ms: Mark-Compact (reduce) 2046.0 (2083.5) -> 2045.5 (2084.3) MB, 623.35 / 0.01 ms  (+ 161.3 ms in 35 steps since start of marking, biggest step 11.5 ms, walltime since start of marking 840 ms) (average mu = 0.422, current [8668:00000294C50F0F90]   149325 ms: Mark-Compact (reduce) 2046.4 (2084.3) -> 2045.6 (2084.5) MB, 1153.60 / 0.00 ms  (average mu = 0.226, current mu = 0.006) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 00007FF63891EEDB node::SetCppgcReference+18123
 2: 00007FF638890C34 DSA_meth_get_flags+89940
 3: 00007FF639307A71 v8::Isolate::ReportExternalAllocationLimitReached+65
 4: 00007FF6392F11C8 v8::Function::Experimental_IsNopFunction+1336
 5: 00007FF639152C70 v8::Platform::SystemClockTimeMillis+659328
 6: 00007FF63914FCF8 v8::Platform::SystemClockTimeMillis+647176
 7: 00007FF63916500A v8::Platform::SystemClockTimeMillis+733978
 8: 00007FF639165887 v8::Platform::SystemClockTimeMillis+736151
 9: 00007FF63916E4BE v8::Platform::SystemClockTimeMillis+772046
10: 00007FF639182CFA v8::Platform::SystemClockTimeMillis+856074
11: 00007FF639182FE3 v8::Platform::SystemClockTimeMillis+856819
12: 00007FF638F117FD v8::base::Thread::StartSynchronously+465613
13: 00007FF638F12897 v8::base::Thread::StartSynchronously+469863
14: 00007FF638F11F54 v8::base::Thread::StartSynchronously+467492
15: 00007FF638E3F4D5 v8::CodeEvent::GetFunctionName+160917
16: 00007FF6393B93FE v8::PropertyDescriptor::writable+678094
17: 00007FF639386168 v8::PropertyDescriptor::writable+468536
18: 00007FF5BAC05A11
'tsc failed. Try cleaning up first: `npm run clean:package-types`'; exit 1
Cannot access this declaration file. You may need to run tsc again: C:\Users\accou\OneDrive\Documentos\gutenberg\packages\core-data\build-types\index.d.ts

Is there any work around? The suggested npm run clean:package-types doesn't resolve this.

sirreal commented 3 months ago

I suspect this is related to some types that are very costly to process. @mirka has landed some work on it here: https://github.com/WordPress/gutenberg/pull/63388

I haven't seen the memory issues myself recently, but a common fix is to increase the amount of memory Node.js can use. That can be handled by an environment variable. On UNIX machines, this would would be an example of how to do that:

NODE_OPTIONS='--max-old-space-size=8000' npm run build:packages

caseymilne commented 3 months ago

I suspect this is related to some types that are very costly to process. @mirka has landed some work on it here: #63388

I haven't seen the memory issues myself recently, but a common fix is to increase the amount of memory Node.js can use. That can be handled by an environment variable. On UNIX machines, this would would be an example of how to do that:
NODE_OPTIONS='--max-old-space-size=8000' npm run build:packages

Good suggestion @sirreal I found this similar command (have not tried it yet) node --max-old-space-size=8000 heapsize.js

caseymilne commented 3 months ago

Not sure if this helps but when I'm running build it's stopping with an error about C:\Users\accou\OneDrive\Documentos\gutenberg\packages\dataviews\build-types\index.d.ts and I tried a suggested command npx ts-node C:\Users\accou\OneDrive\Documentos\gutenberg\packages\dataviews\build-types\index.d.ts only to find that this \packages\dataviews\ does not have a "build-types" directory in it. Maybe a previous step is supposed to create it? There is a \tsconfig.json which has "declarationDir": "build-types" in it. Not sure if \build-types is supposed to be present at the start of the process or get built in an earlier step.

sirreal commented 3 months ago

That directory should be generated during build, but it may not be if an error occurs before. Will you share the git commit sha you're on and the exact command and error you're seeing? That will help to continue debugging.

caseymilne commented 3 months ago

That directory should be generated during build, but it may not be if an error occurs before. Will you share the git commit sha you're on and the exact command and error you're seeing? That will help to continue debugging.

Commit: 154f1aa, command is npm run build, full output below:

> gutenberg@18.8.0 build
> npm run build:packages && wp-scripts build

> gutenberg@18.8.0 prebuild:packages
> npm run clean:packages && lerna run build

> gutenberg@18.8.0 clean:packages
> rimraf "./packages/*/@(build|build-module|build-style)"

lerna notice cli v7.1.4
lerna info versioning independent

    √  @wordpress/block-serialization-spec-parser:build (2s)
    √  @wordpress/babel-preset-default:build (4s)

 ——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  Lerna (powered by Nx)   Successfully ran target build for 2 projects (5s)

> gutenberg@18.8.0 build:packages
> npm run --silent build:package-types && node ./bin/packages/build.js

<--- Last few GCs --->

[3264:000002935424B0F0]   128868 ms: Mark-Compact (reduce) 2045.1 (2083.5) -> 2045.1 (2084.0) MB, 403.79 / 0.01 ms  (+ 281.6 ms in 71 steps since start of marking, biggest step 6.2 ms, walltime since start of marking 791 ms) (average mu = 0.408, current m[3264:000002935424B0F0]   129739 ms: Mark-Compact (reduce) 2046.1 (2084.0) -> 2045.1 (2084.5) MB, 866.64 / 0.00 ms  (average mu = 0.253, current mu = 0.005) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 00007FF670A8EEDB node::SetCppgcReference+18123
 2: 00007FF670A00C34 DSA_meth_get_flags+89940
 3: 00007FF671477A71 v8::Isolate::ReportExternalAllocationLimitReached+65
 4: 00007FF6714611C8 v8::Function::Experimental_IsNopFunction+1336
 5: 00007FF6712C2C70 v8::Platform::SystemClockTimeMillis+659328
 6: 00007FF6712BFCF8 v8::Platform::SystemClockTimeMillis+647176
 7: 00007FF6712D500A v8::Platform::SystemClockTimeMillis+733978
 8: 00007FF6712D5887 v8::Platform::SystemClockTimeMillis+736151
 9: 00007FF6712DE4BE v8::Platform::SystemClockTimeMillis+772046
10: 00007FF6712F2CFA v8::Platform::SystemClockTimeMillis+856074
11: 00007FF6712F2FE3 v8::Platform::SystemClockTimeMillis+856819
12: 00007FF6710817FD v8::base::Thread::StartSynchronously+465613
13: 00007FF671082897 v8::base::Thread::StartSynchronously+469863
14: 00007FF671081F54 v8::base::Thread::StartSynchronously+467492
15: 00007FF670FAF4D5 v8::CodeEvent::GetFunctionName+160917
16: 00007FF6114DAAFA
'tsc failed. Try cleaning up first: `npm run clean:package-types`'; exit 1
Cannot access this declaration file. You may need to run tsc again: C:\Users\accou\OneDrive\Documentos\gutenberg\packages\core-data\build-types\index.d.ts
PS C:\Users\accou\OneDrive\Documentos\gutenberg>

sirreal commented 3 months ago

That looks like a standard out of memory error. The npx ts-node … command isn't going to help here, what may help is setting the environment variable to increase available memory: NODE_OPTIONS='--max-old-space-size=8000'.

Ideally the build would take less memory, but there's still work to be done there 🙂

caseymilne commented 3 months ago

Raising memory didn't work for me, but I've probably only got about 7000 available. Maybe later I'll throw it on a server and see if it can build there.

t-hamano commented 3 months ago

Many of the people reporting issues here seem to be using Windows OS. I'd love to know if the memory exhaustion is happening on the host OS or on WSL (Ubuntu).

By default, WSL has 50% of the memory that the host OS has: https://learn.microsoft.com/en-us/windows/wsl/wsl-config#main-wsl-settings

Therefore, I think we need to check the following:

Are you building on the host OS or on WSL?
How much memory is on the host OS?
How much memory is allocated to WSL?

sirreal commented 1 week ago

This issue is ongoing, there was another report today: https://wordpress.slack.com/archives/C5UNMSU4R/p1728225107128589

benridane commented 1 week ago

I also did a build on a 2G memory VPS and it failed, NODE_OPTIONS='--max-old-space-size=8000' had no effect on 2G.

OS: Ubuntu 22.04.2 LTS memory : 1.9 GiB Swap:2.0 GiB

WordPress / gutenberg