search

apache / kvrocks

Apache Kvrocks is a distributed key value NoSQL database that uses RocksDB as storage engine and is compatible with Redis protocol.
https://kvrocks.apache.org/
Apache License 2.0
3.44k stars 442 forks source link

Evaluate Profile-Guided Optimization on kvrocks #1551

Open zamazan4ik opened 1 year ago

zamazan4ik commented 1 year ago

Search before asking

Motivation

I think Profile-Guided Optimization can improve kvrocks performance. I did a lot of PGO benchmarks on different software (all results are shown here). The most interesting for kvrocks results probably are RocksDB results and Redis results.

Seems like could be a good thing to try on kvrocks as well.

Solution

Try to use PGO and/or BOLT. It could be used via -fprofile-generate/-fprofile-use compiler flags. For further instructions follow GCC or Clang documentation.

Are you willing to submit a PR?

git-hulk commented 1 year ago

@zamazan4ik Thanks for suggesting it, and we definitely like to try any possible improvements. BTW, Kvrocks has applied the LTO/IPO in #764, and looking forward to seeing the result if anyone is interested in this.

aleksraiden commented 1 year ago

So, for implement this feature, we build a first version, run a standard benchmark test, e.g. redis-benchmark and then, rebuild with using a generated gcda file, right?

Or we can static generating this file for general linux and add it into repo (and periodic update)?

zamazan4ik commented 1 year ago

So, for implement this feature, we build a first version, run a standard benchmark test, e.g. redis-benchmark and then, rebuild with using a generated gcda file, right?

Yes. If you want to regenerate profile data per commit or per-optimized build, you can collect .gcda files (in GCC case. In LLVM/Clang case these files will have .profraw format) and then rebuild with these files the project once again.

Or we can static generating this file for general linux and add it into repo (and periodic update)?

You can do it too! Just be careful - in general, these files (.gcda/.profraw) do not preserve compatibility between compilers or even compiler versions. So you will need to maintain these files for multiple compilers (and compiler versions). Or just support only one compiler versions for the optimized build - for other cases describe in the documentation a fallback to a "general" way with rebuilding with Instrumentation, running redis-benchmark, etc.

aleksraiden commented 1 year ago

Lot of thanks, I need a think about this way. You are correct - storing a static file have a bad idea for us, it need to be in CI stage or a dedicated command (or a flag) in x.py build scipt.

raffertyyu commented 7 months ago

Is there any progress? I'm interested in it. Could I have a try to do something about pgo？

git-hulk commented 7 months ago

@raffertyyu Assigned, thank you!