search

rust-lang / rust

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
97.47k stars 12.6k forks source link

Building LLVM from source doesn't forward `-j x` to cmake #95177

Open sassy-crick opened 2 years ago

sassy-crick commented 2 years ago

I am building Rust on a large HPC cluster using EasyBuild but Rust is not behaving as it should:

3450751 pts/1    S+     0:00  |                   \_ /bin/bash /rds/general/user/jsassman/home/.local/bin/eb --robot --download-timeout=100 --modules-tool=EnvironmentModules --module-syntax=Tcl --allow-modules-tool-mismatch -dr --parallel=64 Rust-1.54.0-GCCcore-11.2.0.eb
3450771 pts/1    Sl+    0:04  |                       \_ python3 -m easybuild.main --robot --download-timeout=100 --modules-tool=EnvironmentModules --module-syntax=Tcl --allow-modules-tool-mismatch -dr --parallel=64 Rust-1.54.0-GCCcore-11.2.0.eb

3450859 pts/1    S+     0:00  |                           \_ make -j 64
3450865 pts/1    S+     0:11  |                               \_ /rds/general/user/jsassman/home/apps/easybuild/amd/zen2/software/Python/3.9.6-GCCcore-11.2.0-bare/bin/python3 /dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/bootstrap/bootstrap.py build --stage 2
3452315 pts/1    S+     0:00  |                                   \_ /dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/bootstrap/debug/bootstrap build --stage 2
3487991 pts/1    S+     0:00  |                                       \_ cmake --build . --target install --config Release -- -j 256
3487994 pts/1    S+     0:00  |                                           \_ /usr/bin/gmake -f Makefile -j 256 install

3450751 pts/1    S+     0:00  |                   \_ /bin/bash /rds/general/user/jsassman/home/.local/bin/eb --robot --download-timeout=100 --modules-tool=EnvironmentModules --module-syntax=Tcl --allow-modules-tool-mismatch -dr --parallel=64 Rust-1.54.0-GCCcore-11.2.0.eb
3450771 pts/1    Sl+    0:04  |                       \_ python3 -m easybuild.main --robot --download-timeout=100 --modules-tool=EnvironmentModules --module-syntax=Tcl --allow-modules-tool-mismatch -dr --parallel=64 Rust-1.54.0-GCCcore-11.2.0.eb
--> 3450859 pts/1    S+     0:00  |                           \_ make -j 64
3450865 pts/1    S+     0:11  |                               \_ /rds/general/user/jsassman/home/apps/easybuild/amd/zen2/software/Python/3.9.6-GCCcore-11.2.0-bare/bin/python3 /dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/bootstrap/bootstrap.py build --stage 2
3452315 pts/1    S+     0:00  |                                   \_ /dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/bootstrap/debug/bootstrap build --stage 2
--> 3487991 pts/1    S+     0:00  |                                       \_ cmake --build . --target install --config Release -- -j 256
--> 3487994 pts/1    S+     0:00  |                                           \_ /usr/bin/gmake -f Makefile -j 256 install
3488033 pts/1    S+     0:00  |                                               \_ /usr/bin/gmake -s -f CMakeFiles/Makefile2 all
3488574 pts/1    S+     0:00  |                                                   \_ /usr/bin/gmake -s -f utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/build.make utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/build
3488695 pts/1    S+     0:00  |                                                   |   \_ /rds/general/user/jsassman/home/apps/easybuild/amd/zen2/software/GCCcore/11.2.0/bin/g++ -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/x86_64-unknown-linux-gnu/llvm/build/utils/TableGen -I/dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/llvm-project/llvm/utils/TableGen -I/dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/x86_64-unknown-linux-gnu/llvm/build/include -I/dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/llvm-project/llvm/include -ffunction-sections -fdata-sections -fPIC -m64 -ftree-vectorize -march=native -fno-math-errno -fPIC -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -ffunction-sections -fdata-sections -O3 -DNDEBUG -fno-exceptions -fno-rtti -std=c++14 -MD -MT utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmWriterEmitter.cpp.o -MF CMakeFiles/obj.llvm-tblgen.dir/AsmWriterEmitter.cpp.o.d -o CMakeFiles/obj.llvm-tblgen.dir/AsmWriterEmitter.cpp.o -c /dev/shm/jsassman/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/llvm-project/llvm/utils/TableGen/AsmWriterEmitter.cpp

I expected to see only 64 threads being used but Rust is ignoring my request to use only 64 cores and uses the max-number of available treads. This leads to a very poor performance as each thread spends more time waiting than actually doing. specially when there are 128 threads available. This seems to be a bug in Rust and not in EasyBuild, as above.

Meta

 Rust 1.54.0
krishchanDr commented 2 years ago

Hello, we tried to use PARALLEL_MAKE and tried to achieve the faster build. Here are my observations, 8 Core  - Build completed successfully in 0:22:52 16 Core - Build completed successfully in 0:15:11 32 Core - Build completed successfully in 0:12:37 64 Core - Build completed successfully in 0:12:14 96 Core - Build completed successfully in 0:12:33 128 Core - Build completed successfully in 0:12:11 192 Core - Build completed successfully in 0:12:09 When the PARALLEL_MAKE is greater than or equal to 32, the time remains same... Seems that its run as 32 parallel at the most

Regards Chandra

sassy-crick commented 2 years ago

To add to this: this is roughly what I observe as well: building with cores is not significantly slower than with 64 cores. That said: the problem is that right now it is ignoring the make -j 8 command and uses all available cores, real and virtual ones. That should not happen in my opinion.

jyn514 commented 1 year ago

@sassy-crick it looks like this issue is specifically about cmake not being passed the correct -j flag, is that right? Not cargo in general?

jyn514 commented 1 year ago

Mentoring instructions: Change the code around https://github.com/rust-lang/rust/blob/9ef84076109ca9b6dd87fca7040781d778eb7605/src/bootstrap/native.rs#L315-L317 to pass build_arg(format!("-j{}", builder.config.jobs)) (or configure_arg, I don't know cmake well enough).