search

google / ml-compiler-opt

Infrastructure for Machine Learning Guided Optimization (MLGO) in LLVM.
Apache License 2.0
612 stars 92 forks source link

How to train a model using bin's llvmbc and llvmcmd segments？I want to optimize directly using the executable program #322

Closed 18liumin closed 8 months ago

18liumin commented 9 months ago

/usr1/temp/project/llvm-install/bin/clang -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name b.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -fcoverage-compilation-dir=/usr1/temp/project/ml-compiler-opt/dataset/test_sqrt/build -resource-dir /usr1/temp/project/llvm-install/lib/clang/18 -dependency-file CMakeFiles/test_sqrt.dir/src/b.c.o.d -MT CMakeFiles/test_sqrt.dir/src/b.c.o -sys-header-deps -canonical-system-headers -I /usr1/temp/project/ml-compiler-opt/dataset/test_sqrt/include -c-isystem /usr1/temp/project/tflite/pthreadpool/include -c-isystem /usr/local/include -c-isystem /usr/include -c-isystem /usr1/zwl/software/gcc-12.2.0/build/include -c-isystem /usr/include/c++/11 -cxx-isystem /usr1/temp/project/tflite/pthreadpool/include -cxx-isystem /usr/local/include -cxx-isystem /usr/include -cxx-isystem /usr1/zwl/software/gcc-12.2.0/build/include -cxx-isystem /usr/include/c++/11 -internal-isystem /usr1/temp/project/llvm-install/lib/clang/18/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/11/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir=/usr1/temp/project/ml-compiler-opt/dataset/test_sqrt/build -ferror-limit 19 -fgnuc-version=4.2.1 -fembed-bitcode=all -faddrsig -D __GCC_HAVE_DWARF2_CFI_ASM=1 -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name main.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -fcoverage-compilation-dir=/usr1/temp/project/ml-compiler-opt/dataset/test_sqrt/build -resource-dir /usr1/temp/project/llvm-install/lib/clang/18 -dependency-file CMakeFiles/test_sqrt.dir/src/main.c.o.d -MT CMakeFiles/test_sqrt.dir/src/main.c.o -sys-header-deps -canonical-system-headers -I /usr1/temp/project/ml-compiler-opt/dataset/test_sqrt/include -c-isystem /usr1/temp/project/tflite/pthreadpool/include -c-isystem /usr/local/include -c-isystem /usr/include -c-isystem /usr1/zwl/software/gcc-12.2.0/build/include -c-isystem /usr/include/c++/11 -cxx-isystem /usr1/temp/project/tflite/pthreadpool/include -cxx-isystem /usr/local/include -cxx-isystem /usr/include -cxx-isystem /usr1/zwl/software/gcc-12.2.0/build/include -cxx-isystem /usr/include/c++/11 -internal-isystem /usr1/temp/project/llvm-install/lib/clang/18/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/11/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir=/usr1/temp/project/ml-compiler-opt/dataset/test_sqrt/build -ferror-limit 19 -fgnuc-version=4.2.1 -fembed-bitcode=all -faddrsig -D __GCC_HAVE_DWARF2_CFI_ASM=1 -x ir /tmp/tmp8xuwu4qf/CMakeFiles/test_sqrt.dir/src/test_sqrt/input.bc -mllvm -tfutils-use-simplelogger -mllvm -enable-ml-inliner=development -mllvm -training-log=/tmp/tmp8xuwu4qf/tmpwcc5obsh/log -o /tmp/tmp8xuwu4qf/tmpwcc5obsh/native

error: Invalid encoding 1 error generated.

mtrofin commented 9 months ago

I'd suggest debugging that. Perhaps start with the IR (test_sqrt.dir/src/test_sqrt.o.bc?) Can you llvm-dis it?

18liumin commented 9 months ago

I'd suggest debugging that. Perhaps start with the IR (test_sqrt.dir/src/test_sqrt.o.bc?) Can you llvm-dis it?

I want to combine multiple bc files into one large bc file, and then this large bc file can be compiled into an executable program. Where can I get the parameters needed to compile an executable program?

18liumin commented 9 months ago

I'd suggest debugging that. Perhaps start with the IR (test_sqrt.dir/src/test_sqrt.o.bc?) Can you llvm-dis it?

How to use llvmcmd to compile bc file into an executable program?

mtrofin commented 9 months ago

You'd need to link the .o files produced when compiling your .bc files with the command line in the llvmcmd section (i.e. in the .cmd files in the corpus). Find the linker command line from your build system.

I may be able to help better if I understood what the goal is for getting to an executable this way; for example, if it's to try out a policy, you could just compile with that policy the whole project by adding the necessary compilation flags.

18liumin commented 9 months ago

You'd need to link the .o files produced when compiling your .bc files with the command line in the llvmcmd section (i.e. in the .cmd files in the corpus). Find the linker command line from your build system.

I may be able to help better if I understood what the goal is for getting to an executable this way; for example, if it's to try out a policy, you could just compile with that policy the whole project by adding the necessary compilation flags.

I want to directly measure the performance of the optimized executor during training. -o /tmp/tmp_fxu6qar/tmpu9tuxv8a/native, i want to test native's performance

mtrofin commented 9 months ago

Got it, so ya, you need to figure out your linker command line.