Document memory consumption/best practices for build [WAS: Out of heap space?]

CntrlX commented 3 years ago

16 GB ram not enough for compilation ?

E:\o3de-build\External\LyShine-67f2e481\Code\CMakeFiles\LyShine.Editor.Static.dir\Unity\unity_10_cxx.cxx(19,1): fatal e rror C1060: compiler is out of heap space [E:\o3de-build\External\LyShine-67f2e481\Code\LyShine.Editor.Static.vcxproj] E:\o3de-cache\packages\qt-5.15.2-rev4-windows\qt\include\QtCore/qmetatype.h(1157,1): fatal error C1060: compiler is out of heap space [E:\o3de-build\External\PhysX-c771218d\Code\PhysX.Editor.Static.vcxproj]

Originally posted by @seccentral in https://github.com/o3de/o3de/discussions/2077

nemerle commented 3 years ago

I've seen the same kind of problem when compiling on a 24-core machine. Basically, VS2019 is trying to start 24 compilation processes, which needs quite a lot of memory. You could try reducing the number of parallel jobs in VS options ( it helped a bit here)

jspanchu commented 3 years ago

Can confirm the same on a qemu windows VM (8 vCPU, 16GB mem). I got it to work by turning of unity build.

You could add -DLY_UNITY_BUILD=OFF to the cmake.exe configure command line or do it from cmake-gui.exe

amznestebanpapp commented 3 years ago

Our measurements of memory usage is around 2Gb per core. So, if you have 8 cores, you should have at least 16Gb of RAM (available). If you don't have enough, here is a list of things to tweak:

tweak your page file, increase and make it at least twice your amount of RAM
reduce the amount of "project parallelization", instead of passing /m, pass something like /m:2, that will run just 2 projects at the same time. The heaviest memory consumption comes from linking big libraries, which happens once per project, therefore tweaking this setting should reduce your memory usage.
reduce the amount of "parallel compilation within the project", add /p:CL_MPCount=4 (after the /m:2), that will run two compilations within the project at the same time.
remove test targets from the build, pass -DLY_DISABLE_TEST_MODULES=1 when generating the project/solution. This will reduce the amount of targets, but those targets dont consume lots of memory. This is mainly to reduce compilation time.

The items above will cause slower builds.

We have been talking with some people and it seems that we will be changing the documentation to add a "recommended" spec. If your spec is bellow the recommended, we will add these tweaks to the docs.

amznestebanpapp commented 3 years ago

o3de / o3de.org

Document memory consumption/best practices for build [WAS: Out of heap space?] #949