search

ambaxter / bbolt-rs

Port of bbolt in Rust
Other
63 stars 3 forks source link

Evaluate using Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO) #2

Open zamazan4ik opened 1 month ago

zamazan4ik commented 1 month ago

Hi!

Recently I checked many optimizations like PGO and PLO (mostly with LLVM BOLT) improvements on multiple projects. The results are available here. According to the tests, these optimizations can help to achieve better performance in many cases like databases. I think trying to optimize this project with them will be an interesting idea to achieve more performance.

I already did some (very basic!) benchmarks and want to share my results here.

Test environment

Benchmark

For benchmark purposes, I used these benchmarks. The PGO training workload was bench command run. The release and PGO-optimized results are generated with bench -c 10000000.

All PGO and PLO optimizations are done with cargo-pgo. All tests are done on the same machine, done multiple times, with the same background "noise" (as much as I can guarantee of course) - the results are consistent enough across runs. taskset -c 0 is used for reducing the OS scheduler result interference.

Results

Let's start with the results.

Release:

taskset -c 0 bench -c 10000000
# Write  8.914817311s  (891ns/op)  (1122334 op/sec)
# Read  1.11635498s  (16ns/op)  (62499999 op/sec)

Release + PGO optimization:

taskset -c 0 bench -c 10000000
# Write  6.502718524s  (650ns/op)  (1538461 op/sec)
# Read  1.06592023s  (13ns/op)  (76923076 op/sec)

Release + PGO optimization + BOLT optimization:

taskset -c 0 bench -c 10000000
# Write  6.50363167s  (650ns/op)  (1538461 op/sec)
# Read  1.0854954s  (14ns/op)  (71428571 op/sec)

(just for reference) Release + PGO instrumentation:

taskset -c 0 bench -c 10000000
# Write 14.626009666s   (1.463µs/op)    (683526 op/sec)
# Read  1.131095569s    (28ns/op)   (35714285 op/sec)

(just for reference again) Release + PGO optimized + BOLT instrumented:

taskset -c 0 bench -c 10000000
# Write  8.641237686s  (864ns/op)  (1157407 op/sec)
# Read  1.102478063s  (12ns/op)  (83333333 op/sec)

According to the tests above, I see measurable improvements from enabling PGO in performance. However, enabling PLO with LLVM BOLT didn't show measurable improvements at least in the simple test above.

For anyone interested in binary sizes, I collected some statistics too (without debug symbols stripping):

The only interesting case here is the last one - "Release + PGO optimized + BOLT optimized". I don't know why the binary size was increased so much. I guess some "magic" BOLT's option should be involved here and "fix" the situation. However, it's just a guess for now, no more.

Further steps

I can suggest the following action points:

Here are some examples of how PGO optimization is integrated into other projects:

I would be happy to answer your questions about all the optimizations above. Please do not treat the issue as a bug or something like that - it's just an idea of how the project performance can be improved.

ambaxter commented 1 month ago

Hi @zamazan4ik ,

I played with PGO about a month ago, but decided to wait until after I had completed the first release to take a deeper look.

Do you have an updated link to the Rust ci script? It currently points to a 404.

Also, would you mind describing the 2nd step a bit more? I'm not sure what you mean.

zamazan4ik commented 1 month ago

Do you have an updated link to the Rust ci script? It currently points to a 404.

Oh, sorry for that - the tool is changed in the upstream and now it's rewritten in Rust from Python: https://github.com/rust-lang/rust/tree/master/src/tools/opt-dist . Updated the original post as well.

Also, would you mind describing the 2nd step a bit more? I'm not sure what you mean.

Sure! By "Providing an easier way (e.g. a build option) to build scripts with PGO" I mean extending existing bbolt-rs's build infra (scripts) with an additional option - building the database with PGO. It can look like make build_with_pgo (just an example!). With one simple command, it can be easier for users to build their own bbolt-rs version and tweak it accordingly to their workloads. Is it worth it or not - it's up to you. My 0.5$ - it's not worth it on the current project lifecycle stage.