Evaluate Profile-Guided Optimization (PGO) and LLVM BOLT

[zamazan4ik]

Description

Recently I checked Profile-Guided Optimization (PGO) improvements on many projects. All current results are available here. E.g. ClickHouse PGO results can be checked here. According to the multiple tests, PGO can help with improving performance in many cases. That's why I think trying to optimize the Gluten with PGO can be a good idea.

I can suggest the following action points:

• Perform PGO benchmarks on Gluten. And if it shows improvements - add a note about possible improvements in Gluten performance with PGO.
• Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users and maintainers since they will be able to optimize Gluten according to their own workloads.
• Optimize pre-built binaries

Since the Gluten native part is the library, I think the Pydantic-core experience can be reused here. Also, Clang supports PGO for shared libraries.

Maybe testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

Here are some examples of how PGO optimization is integrated in other projects:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag

[baibaichen]

@zhanglistar

[FelixYBW]

It's good idea to have such compiler optimizations. The problem is which workload should be used to profile. TPCH/DS doesn't match customers' real workload.

[zamazan4ik]

The problem is which workload should be used to profile. TPCH/DS doesn't match customers' real workload.

If you say that TPCH/DS does not match customers' real workload, that means you know the actual workload for your customers, right? :)

One of the options could be:

• Performing the PGO benchmarks on some as much as possible real-life workload from your point of view
• Document the benchmark results somewhere in the Gluten documentation
• Provide an option in the build scripts to build Gluten with PGO

In this case, you have no need to guess about the customers' workload. For generic customers, you will have a "default" build (as you already provide), but for customers who want to extract as much as possible performance - they will have a ready-to-use build option in Gluten to recompile the library according to the customer's workload.

If you still want to optimize prebuilt binaries with PGO, you can try to collect feedback from your users about their actual workloads via discussions, collecting their PGO profiles, etc. Then all collected information can be used for making a "generic enough" profile that can be used in Gluten CI for building a PGO-optimized binary.