search

KhronosGroup / SPIRV-LLVM-Translator

A tool and a library for bi-directional translation between SPIR-V and LLVM IR
Other
478 stars 213 forks source link

Reduce memory usage during compilation #412

Open KOLANICH opened 4 years ago

KOLANICH commented 4 years ago

SPIRV-memory_consumption

The flags were ninja -j 1 -k 0, so it is a single thread consumption and if an attempt had failed, the build system starts building the next cpp file.

The compiler is g++-9, the build type is Release.

The plot is made by using psrecord and then importing the file into python using np.genfromtxt and plotting using matplotlib.pyplot.

When compiling this lib compilation often fails even on PCs with 8 GiB of RAM (a guaranteed way to fail it is to just run Firefox in background). Overcommit is disabled to mitigate complete system hangs because of 12309.

Also if always fails without -j 1 when building from scratch (one has to run ninja multiple times to get it built).

So I feel like that memory consumption should be reduced somehow.

AlexeySachkov commented 4 years ago

@KOLANICH, are you using pre-built LLVM? Which build type are you using?

I wouldn't expect the translator library to consume 8 GiB of memory during compilation, but this might be true for LLVM.

Just tried on my machine: I have CMAKE_BUILD_TYPE=Debug and pre-built versions of LLVM & Clang. I've launched make llvm-spirv -j1 and watched to htop output - the biggest value of memory consumption I saw is around 2GiB (and it seems that ~700 MiB are allocated for some other processes launched on my machine)

KOLANICH commented 4 years ago

are you using pre-built LLVM? Which build type are you using?

Yes, llvm-10 from the official deb repo. Release build type.

MrSidims commented 6 months ago

FYI @bwlodarcz