search

substrait-io / substrait-java

Apache License 2.0
75 stars 72 forks source link

Calcite optimizations in Isthmus/Substrait plans #78

Open Advitya17 opened 2 years ago

Advitya17 commented 2 years ago

Based on our experiments with Substrait plans and conversations with the Substrait folks highlighted on the Google Group, it looks like the Substrait plans currently generated by Isthmus are unoptimized.

We're actively looking to use Substrait plans infused with Calcite optimizations for our internal projects. Please keep us updated on when support for the same could be added, so that we can incorporate and experiment with the optimized plans.

Additionally, if there're any other (ideally pythonic) tools/APIs that successfully generate optimized Substrait plans (from any other query optimizers) now or in the future, please let us know and we'd love to utilize them for our projects.

Thank you very much for the continued support!

jacques-n commented 2 years ago

While it is true that the current Isthmus tool doesn't do any optimization, it would likely be trivial to enable a number of optimizations as an optional behavior in Isthmus. This has always been the intention. Would you like to propose a patch which adds a new flag that also causes Isthmus to run through a set of optimizations. Maybe @hbutani or @jcamachor would be willing to help guide you.

Calcite has a rich set of pre-packaged optimizations we could use. The only optimization types that we'd probably have to disable are any that use runtime code generation (since that doesn't work in Graal). I think that is constrained to the Calcite constant reduction interpreter. (I could be wrong about it requiring runtime code generation.)

adamkennedy commented 1 year ago

constant reduction, and some similar things like Values evaluation through Project/Filter/etc pretty much all use code generation.