Open j2kun opened 4 months ago
j2kun
commented
3 months ago My system:
12th Gen Intel(R) Core(TM) i9-12900K 3.20 GHz, 64.0 GB RAM (24 cores) Ubuntu clang version 17.0.6 (++20231209124227+6009708b4367-1~exp1~20231209124336.77)
From a clean build (bazel clean --expunge), bazel-build ...:all takes 27m 21s
j2kun
commented
3 months ago Note taking as I go:
There are some dependencies that seem unnecessary, which I might be able to clean up upstream. For example,
bazel query 'somepath(//tools:heir-opt, @llvm-project//llvm:include/llvm/BinaryFormat/ELFRelocs/Lanai.def)'
//tools:heir-opt
@llvm-project//mlir:LLVMDialect
@llvm-project//llvm:BinaryFormat
@llvm-project//llvm:include/llvm/BinaryFormat/ELFRelocs/Lanai.defLLVM dialect pulls in basically all of LLVM, even though it doesn't seem reasonable to run LLVM analyses themselves on the LLVM MLIR dialect.
bazel query 'somepath(//tools:heir-opt, @llvm-project//mlir:GPUDialect)'
//tools:heir-opt
@llvm-project//mlir:LinalgTransforms
@llvm-project//mlir:GPUDialectLots of upstream dialect dependencies get pulled in through LinalgTransforms, including GPU and Mesh and SparseTensor, while things like NVGPU get pulled in through MemRefTransforms. Might be able to separate them into smaller build targets to avoid.
j2kun
commented
3 months ago After https://github.com/google/heir/pull/895, the build time does not meaningfully change.
j2kun
commented
3 months ago One thing I'm trying to do, but haven't quite figured out how to do, is to run a profile of the build to get some sort of idea of what takes up most of the build time.
j2kun
commented
3 months ago Trying again, but more specific (requires bazel version 7.0+):
bazel clean --expunge
bazel build --profile build.profile //tools:heir-opt
bazel analyze-profile build.profile=== PHASE SUMMARY INFORMATION ===
Total launch phase time 0.694 s 0.10%
Total init phase time 79.458 s 11.62%
Total target pattern evaluation phase time 0.027 s 0.00%
Total interleaved loading-and-analysis phase time 7.000 s 1.02%
Total preparation phase time 0.014 s 0.00%
Total execution phase time 596.347 s 87.24%
Total finish phase time 0.027 s 0.00%
---------------------------------------------------------------------
Total run time 683.570 s 100.00%
Critical path (105.226 s):
Time Percentage Description
32.7 ms 0.03% action 'Executing genrule @llvm-project//llvm generate_static_extension_registry [for tool]'
0.06 ms 0.00% cc_library-compile for @llvm-project//llvm Support
0.12 ms 0.00% cc_library-compile for @llvm-project//llvm TableGen
6.111 s 5.81% action 'Compiling llvm/lib/TableGen/Record.cpp [for tool]'
84.5 ms 0.08% action 'Linking external/llvm-project/llvm/llvm-min-tblgen [for tool]'
0.51 ms 0.00% runfiles for @llvm-project//llvm llvm-min-tblgen
2.028 s 1.93% action 'Generating code from table lib/Target/RISCV/RISCV.td @llvm-project//llvm RISCVTargetParserDefGen__gen_riscv_target_def_genrule [for tool]'
0.05 ms 0.00% cc_library-compile for @llvm-project//llvm TargetParser
0.05 ms 0.00% cc_library-compile for @llvm-project//mlir TableGen
0.01 ms 0.00% cc_library-compile for @llvm-project//mlir MlirTableGenMain
0.08 ms 0.00% BazelCppSemantics_build_arch_k8-opt-exec-2B5CBBC6 for @llvm-project//mlir mlir-tblgen
16.928 s 16.09% action 'Compiling mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp [for tool]'
99.1 ms 0.09% action 'Linking external/llvm-project/mlir/mlir-tblgen [for tool]'
0.13 ms 0.00% runfiles for @llvm-project//mlir mlir-tblgen
75.2 ms 0.07% action 'TdGenerate external/llvm-project/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.cpp.inc'
0.03 ms 0.00% cc_library-compile for @llvm-project//mlir LLVMIntrinsicOpsIncGen
0.05 ms 0.00% cc_library-compile for @llvm-project//mlir LLVMDialect
74.724 s 71.01% action 'Compiling mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp'
5.143 s 4.89% action 'Linking tools/heir-opt'
0.39 ms 0.00% runfiles for //tools heir-opt=== PHASE SUMMARY INFORMATION ===
Total launch phase time 0.785 s 0.19%
Total init phase time 89.419 s 21.97%
Total target pattern evaluation phase time 0.050 s 0.01%
Total interleaved loading, analysis and execution phase time 316.673 s 77.81%
Total finish phase time 0.032 s 0.01%
---------------------------------------------------------------------
Total run time 406.960 s 100.00%
Critical path (83.635 s):
Time Percentage Description
1.80 ms 0.00% action 'Writing script external/llvm-project/mlir/MlirTableGenMain.cppmap [for tool]'
16.740 s 20.02% action 'Compiling mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp [for tool]'
138 ms 0.17% action 'Linking external/llvm-project/mlir/mlir-tblgen [for tool]'
0.12 ms 0.00% runfiles for @@llvm-project//mlir mlir-tblgen
219 ms 0.26% action 'TdGenerate external/llvm-project/mlir/include/mlir/Interfaces/MemorySlotOpInterfaces.h.inc'
64.244 s 76.81% action 'Compiling mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp'
2.291 s 2.74% action 'Linking tools/heir-opt'
0.36 ms 0.00% runfiles for //tools heir-optSo it looks like while the critical path was not significantly reduced, there is less work being done in total (683s vs 406s), so I'm not doing nothing! There are probably other targets that need to be pruned.
j2kun
commented
3 months ago For bazel build --profile build_test.profile tests/...:all
The critical path is only like 15% of the total runtime... gotta see what else is being built.
=== PHASE SUMMARY INFORMATION ===
Total launch phase time 0.045 s 0.00%
Total init phase time 0.130 s 0.01%
Total target pattern evaluation phase time 0.005 s 0.00%
Total interleaved loading-and-analysis phase time 7.920 s 0.77%
Total preparation phase time 0.092 s 0.01%
Total execution phase time 1019.486 s 99.20%
Total finish phase time 0.031 s 0.00%
---------------------------------------------------------------------
Total run time 1027.711 s 100.00%
Critical path (141.435 s):
Time Percentage Description
0.32 ms 0.00% action 'Writing script external/llvm_zstd/zstd.cppmap [for tool]'
22.6 ms 0.02% action 'Compiling llvm/lib/Support/SlowDynamicAPInt.cpp [for tool]'
0.60 ms 0.00% action 'Linking external/llvm-project/llvm/llvm-min-tblgen [for tool]'
0.10 ms 0.00% runfiles for @llvm-project//llvm llvm-min-tblgen
1.123 s 0.79% action 'Generating code from table include/llvm/IR/Intrinsics.td @llvm-project//llvm intrinsic_S390_gen__gen_intrinsic_enums__intrinsic_prefix_s390_genrule [for tool]'
0.01 ms 0.00% cc_library-compile for @llvm-project//llvm Core
0.01 ms 0.00% cc_library-compile for @llvm-project//llvm BitReader
0.00 ms 0.00% cc_library-compile for @llvm-project//llvm IRReader
0.00 ms 0.00% cc_library-compile for @llvm-project//llvm Object
0.01 ms 0.00% cc_library-compile for @llvm-project//llvm DebugInfo
0.00 ms 0.00% cc_library-compile for @llvm-project//llvm DebugInfoBTF
0.00 ms 0.00% cc_library-compile for @llvm-project//llvm DebugInfoPDB
0.00 ms 0.00% cc_library-compile for @llvm-project//llvm Symbolize
0.00 ms 0.00% cc_library-compile for @llvm-project//llvm ProfileData
0.00 ms 0.00% cc_library-compile for @llvm-project//llvm Analysis
0.00 ms 0.00% cc_library-compile for @llvm-project//llvm BitWriter
0.01 ms 0.00% cc_library-compile for @llvm-project//mlir LLVMDialect
128.406 s 90.79% action 'Compiling mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp [for tool]'
1.594 s 1.13% action 'Linking tools/heir-opt [for tool]'
0.31 ms 0.00% runfiles for //tools heir-opt
7.745 s 5.48% action 'Action tests/openfhe/end_to_end/box_blur_64x64_test_heir_opt.mlir'
5.54 ms 0.00% action 'Action tests/openfhe/end_to_end/box_blur_64x64_lib.h'
0.03 ms 0.00% cc_library-compile for //tests/openfhe/end_to_end box_blur_64x64_test_cc_lib
2.437 s 1.72% action 'Compiling tests/openfhe/end_to_end/box_blur_64x64_test_lib.inc.cc'
48.2 ms 0.03% action 'Linking tests/openfhe/end_to_end/libbox_blur_64x64_test_cc_lib.so'
0.28 ms 0.00% action 'SolibSymlink _solib_k8/libtests_Sopenfhe_Send_Uto_Uend_Slibbox_Ublur_U64x64_Utest_Ucc_Ulib.so'
52.3 ms 0.04% action 'Linking tests/openfhe/end_to_end/box_blur_64x64_test'
0.06 ms 0.00% runfiles for //tests/openfhe/end_to_end box_blur_64x64_testIt looks like the main extra things built in the test invocation are test-critical things: openfhe and its dependencies, googletest, absl, rapidjson, and the heir-translate emitters.
j2kun
commented
3 months ago Ah, here we go! We have a dependency of mlir-opt in our tests!
bazel query 'somepath(//tests:all, @llvm-project//llvm:AArch64CodeGen)'
//tests:test_utilities
@llvm-project//mlir:mlir-opt
@llvm-project//llvm:AllTargetsCodeGens
@llvm-project//llvm:AArch64CodeGenThis is incurring a huge bloat from building all codegen targets, etc., when running tests (but not heir-opt).
We do use these, but we shouldn't anymore
$ rg mlir-opt tests
tests/memref_global_raw.mlir
10:// RUN: mlir-opt %s -pass-pipeline="builtin.module( \
27:// RUN: mlir-opt -pass-pipeline="builtin.module( \
tests/memref_global.mlir
8:// RUN: mlir-opt %s -pass-pipeline="builtin.module( \
25:// RUN: mlir-opt -pass-pipeline="builtin.module( \
tests/BUILD
25: "@llvm-project//mlir:mlir-opt",I was able to remove mlir-opt, which helped (in particular, removing all the code in mlir/test from the dependencies), but it seems the next hurdle is that a lot of LLVM code is pulled in due to our use of mlir-cpu-runner, which is a bit more widespread.
j2kun
commented
3 months ago As of now, https://github.com/google/heir/pull/895 has a wallclock time of 25 minutes, which is... an improvement I guess.
j2kun
commented
3 months ago Without removing the use of mlir-cpu-runner, I don't think we'll be able to do much more here. Closing.
AlexanderViand-Intel
commented
1 month ago Re-opening this and adding "reducing build (disk) size" as a goal, as that's the sticking point for devcontainer.json/ default GH codespaces (see #1001).
Building HEIR+LLVM using bazel requires around 25GB of disk space (that's probably in DEBUG mode) (See https://github.com/google/heir/pull/1001#issuecomment-2394912055)
AlexanderViand-Intel
commented
1 month ago Looking through the bazel llvm setup, I noticed this comment. @j2kun any update on removing the NVPTX target?
Also, I think it'd make sense to replace X86 and AArch64 with the special Native target.
https://github.com/google/heir/blob/94fae4f9a0ea1bc73319cf5d5eb8008fb02b97d4/bazel/setup_llvm.bzl#L8-L17
AlexanderViand-Intel
commented
1 month ago Also, where do we specify which parts of LLVM we want?
I.e., what's the bazel equivalent of -DLLVM_ENABLE_PROJECTS=mlir?
j2kun
commented
1 month ago Bazel just defines build targets and figures out all the dependencies you need when you attempt to build a target. I think one of the issues here is that some of our targets (e.g., the test targets that depend on mlir-cpu-runner) pull in a lot of upstream LLVM that is otherwise unused by HEIR. This can caused by upstream bazel build target definitions pulling in more dependencies than they strictly need, though when I looked into it I don't think they do that, outside of the dependency of some tests on mlir-cpu-runner.
The MLIR build time has a lot of stuff for code gen for various architectures that we don't target (e.g., SPIRV) and passes and dialects that we don't use (e.g., gpu, arm*, etc.). We could likely save a lot of build time by opting-in to including just the dialects and passes we need in heir-opt, and that would also likely cut down the size of the final built binary.