Compile apollo using wllvm

[E00N777]

I want to use llvm to compile apollo (no matter which version), but apollo uses bazel. I tried to modify the .bazelrc and .apollo.bazelrc and tools/bazel.rc files to change bazel's default compiler to llvm, but I still encountered Got this error: (04:48:24) ERROR: /apollo/.cache/bazel/540135163923dd7d5820f3ee4b306b32/external/zlib/BUILD.bazel:31:11: C++ compilation of rule '@zlib//:zlib' failed (Exit 1): wllvm failed: error executing command (cd /apollo/.cache/bazel/540135163923dd7d5820f3ee4b306b32/execroot/apollo && \ exec env - \ LD_LIBRARY_PATH=/usr/local/libtorch_gpu/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 \ PATH=/apollo/modules/teleop/common/scripts:/apollo/bazel-bin/modules/tools/visualizer:/apollo/bazel-bin/cyber/tools/cyber_launch:/apollo/bazel-bin/cyber/tools/cyber_service:/apollo/bazel-bin/cyber/tools/cyber_node:/apollo/bazel-bin/cyber/tools/cyber_channel:/apollo/bazel-bin/cyber/tools/cyber_monitor:/apollo/bazel-bin/cyber/tools/cyber_recorder:/apollo/bazel-bin/cyber/mainboard:/usr/local/cuda/bin:/opt/apollo/neo/bin:/opt/apollo/sysroot/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ PWD=/proc/self/cwd \ /usr/local/bin/wllvm -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -MD -MF bazel-out/host/bin/external/zlib/_objs/zlib/uncompr.pic.d '-frandom-seed=bazel-out/host/bin/external/zlib/_objs/zlib/uncompr.pic.o' -fPIC -iquote external/zlib -iquote bazel-out/host/bin/external/zlib -isystem external/zlib/zlib/include -isystem bazel-out/host/bin/external/zlib/zlib/include -g0 -mavx2 -Wno-unused-variable -Wno-implicit-function-declaration -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/zlib/uncompr.c -o bazel-out/host/bin/external/zlib/_objs/zlib/uncompr.pic.o) Execution platform: @local_config_platform//:host CRITICAL: No compiler set. Please set environment variable LLVM_COMPILER I tried export LLVM_COMPILER=clang but it's uselness.

Do I need to redefine the tool chain?

Has anyone tried using llvm to compile apollo and extract the bitcode? I need help

[lilejin322]

Apollo is compiled by GCC-7 by default. Here is a repository based on Baidu Apollo 3.0 and 5.0 that uses LLVM for compilation, but I'm not sure if it's still usable. Currently, a Ph.D. student from Nanjing University, China and I have been unable to reproduce the build, and we don't know where the problem lies. It doesn't directly specify Clang in the Bazel script but instead uses the python wllvm package https://github.com/travitch/whole-program-llvm, to extract the LLVM Intermediate Representation for the purpose of static analysis (model checking). https://github.com/h1994st/ApolloLLVMCompile Anyone has suggestions?

[E00N777]

don't use wllvm，just use llvm compile apollo directly. if you need instrumentation when you are compiling，you have to combine your pass into llvm. don't use old version llvm and apollo，there will be a lot of problems. I have successfully compiled apollo 7.0 with bazel and llvm(maybe6.0 I forget)

On Tue, Aug 13, 2024, 4:22 PM Lejin Li @.***> wrote:

Here is a repository based on Baidu Apollo 3.0 and 5.0 that uses LLVM for compilation, but I'm not sure if it's still usable. Currently, a Ph.D. student from Nanjing University, China and I have been unable to reproduce the build, and we don't know where the problem lies. This guide is quite strange; it doesn't directly specify Clang in the Bazel script but instead uses the python wllvm package, to extract the LLVM Intermediate Representation for the purpose of static analysis (model checking). https://github.com/h1994st/ApolloLLVMCompile Anyone has suggestions?

— Reply to this email directly, view it on GitHub https://github.com/ApolloAuto/apollo/issues/15199#issuecomment-2285653905, or unsubscribe https://github.com/notifications/unsubscribe-auth/BDKQFSCCN6UWW7UYPIYUS7LZRG65BAVCNFSM6AAAAABMNXRROCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBVGY2TGOJQGU . You are receiving this because you authored the thread.Message ID: @.***>

[lilejin322]

don't use wllvm，just use llvm compile apollo directly. if you need instrumentation when you are compiling，you have to combine your pass into llvm. don't use old version llvm and apollo，there will be a lot of problems. I have successfully compiled apollo 7.0 with bazel and llvm(maybe6.0 I forget)

While I've tried to define the CC-toolchain using wllvm in new version of Apollo (r7.0.0), implement of my code available as follow, but it doesn't work.

If I use LLVM for compilation directly, I’m not sure how to modify the Bazel scripts for the entire stack. It seems that CUDA-related modules will raise Errors as long as compiler frontend is specified as Clang in Bazel config files. If you only modify the compiler priority via shell in Docker, it won’t work because Bazel will still instruct GCC to handle the entire compilation process. This can be verified by grep 'GCC' tag from the bazel-bin files.

[Update]

To further clarify the effort to introduce cc-toolchain into Apollo 7.0, I forked a public Apollo code repository and started from version r7.0.1, which has already fixed the zlib bug: https://github.com/YuqiHuai/BaiduApollo/tree/r7.0.1

Showcasing all my changes (to simplify the process, I only created a single commit): https://github.com/lilejin322/BaiduApollo/tree/r7.0.1_debug

Here is an only-one-click comparision of all code changes based on branch --r7.0.1 from a public repository https://github.com/YuqiHuai/BaiduApollo/compare/r7.0.1...lilejin322:BaiduApollo:r7.0.1_debug This attempt still has problems, this phenomenon can be directly reproduced. 👇

In host linux:

HOST$ git clone https://github.com/lilejin322/BaiduApollo.git -b r7.0.1_debug
HOST$ cd apollo
HOST/apollo$ bash docker/scripts/dev_start.sh
HOST/apollo$ bash docker/scripts/dev_into.sh

In docker container:

INDOCKER/apollo$ pip install wllvm    # install wllvm, here pip from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)
INDOCKER/apollo$ bash apollo.sh build

Then, you will get the reproduced error.

(17:48:46) ERROR:
/apollo/.cache/bazel/540135163923dd7d5820f3ee4b306b32/external/com_google_protobu
f/BUILD:161:11: undeclared inclusion(s) in rule '@com_google_protobuf//:protobuf_lite':
this rule is missing dependency declarations for the following files included by
'com_google_protobuf/src/google/protobuf/stubs/stringprintf.cc':
'/usr/lib/clang/10.0.0/include/stdarg.h'
'/usr/lib/clang/10.0.0/include/stddef.h'
'/usr/lib/clang/10.0.0/include/stdint.h'
'/usr/lib/clang/10.0.0/include/__stddef_max_align_t.h'
'/usr/lib/clang/10.0.0/include/limits.h'
(17:48:46) INFO: Elapsed time: 15.665s, Critical Path: 0.68s
(17:48:46) INFO: 23 processes: 21 internal, 2 local.
(17:48:46) FAILED: Build did NOT complete successfully

This is a really weird phenomenon 😂 Coz I've declared the position of /usr/bin/clang/10.0.0/include in the CROSSTOOL: https://github.com/lilejin322/BaiduApollo/blob/d13c1999ce7d13fa2fd907365fc43739a3395db2/tools/wllvm/CROSSTOOL#L53

I can 100% confirm these head files exist. :pleading_face:

• P.S. I'm an amateur in this field. 👓

[Motivation] Q: Why use LLVM for compilation instead of GCC? A: LLVM provides powerful intermediate representations like bitcode (.bc) and human-readable intermediate representation (.ll) files. These representations are derived from high-level programming languages and are logically equivalent to them. Analyzing compiler intermediate representations helps in understanding the inherent logic of a program, rather than relying solely on fuzz testing (black-box testing).

Q: How to use these intermediate representations (IR) ? A: Intermediate representations can be used to generate Control Flow Graphs (CFG), Control Dependence Graphs (CDG), and Program Dependence Graphs (PDG). These graphs are extremely useful for identifying potential logical errors within a program. One available toolkit: https://github.com/ARISTODE/program-dependence-graph

• I strongly recommend successors who want to tackle this topic should read the following references. ⬇️

[References] 📖

• History Contributions [1] https://github.com/ZiwenWan/wllvm-toolchain/tree/master [2] https://github.com/sera-sch/apollo-clang-toolchain [3] https://github.com/baiduxlab/apollo/tree/master/tools/clang-6.0 [4] https://github.com/h1994st/ApolloLLVMCompile [5] https://github.com/zqzqz/AVChecker/tree/master/apollo_llvm_compile [6] Bazel Tutorial: Configure C++ Toolchains [7] https://github.com/vsco/bazel-toolchains
• New features regarding Bazel Build System [8] Bazel Migrating to Platforms
• One possible solution [9] https://github.com/bazel-contrib/toolchains_llvm

[Appendix] Bazel version comparison between Apollo 5.0 official docker image and Apollo 7.0 official docker image.

• v 5.0
• v 7.0