search

lakehq / sail

LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.
https://lakesail.com
Apache License 2.0
524 stars 13 forks source link

Evaluate using Profile-Guided Optimization (PGO) for the library #193

Open zamazan4ik opened 2 months ago

zamazan4ik commented 2 months ago

Hi!

Recently I did many Profile-Guided Optimization (PGO) benchmarks on multiple projects (including many parsers and databases) - the results are available here. I think such optimization can be useful for Sail too.

We need to evaluate PGO applicability to Sail. If it helps to achieve better performance - add a note to the documentation about that. In this case, users and maintainers will be aware of another optimization opportunity for Sail. Also, PGO integration into the build scripts can help users and maintainers easily apply PGO to their workloads. Even distributed by Sail binaries (wheels) can be pre-optimized with PGO on a generic-enough sample workload (e.g. rustc already does it).

After PGO, I can suggest evaluating LLVM BOLT as an additional optimization step. For the Rust projects, I recommend starting with cargo-pgo - it makes easier PGO optimization in many cases. Since Sail also has a Python wheel form, I also can recommend taking a look at how PGO is already integrated into other Rust + Python projects like Pydantic: https://github.com/pydantic/pydantic-core/pull/741

Of course it all make sense if you care about Sail's CPU performance :)

Thank you.

P.S. It's not an actual issue - just an improvement idea. I created the issue since Discussions are disabled for the repository.

shehabgamin commented 2 months ago

Thank you for sharing your insights and these valuable resources! We truly appreciate your interest in enhancing Sail's performance.

I've briefly explored PGO in the past, and I agree that evaluating its applicability to Sail is a great idea. If you're interested, we'd welcome your contribution to explore PGO integration (and any of your other suggestions) with Sail. Your expertise would be incredibly helpful.