search

oap-project / gazelle_plugin

Native SQL Engine plugin for Spark SQL with vectorized SIMD optimizations.
Apache License 2.0
256 stars 77 forks source link

[NSE-1135] Use spark-3.2.1 shim layer for spark-3.2.2 #1154

Closed PHILO-HE closed 1 year ago

PHILO-HE commented 1 year ago

What changes were proposed in this pull request?

After cleaning up some code in a previous patch, it looks spark321 shim layer can be used into spark322, i.e., building with spark-3.2 or spark-3.2.1 profile, then using the generated jar for spark322. This patch is experimental and needs sufficient tests.

How was this patch tested?

Benchmark tests on spark-322.

github-actions[bot] commented 1 year ago

https://github.com/oap-project/native-sql-engine/issues/1135

PHILO-HE commented 1 year ago

TPC-DS test passed. @jackylee-ch, please have some tests with this patch. Please note you should use -Pspark-3.2 or -Pspark-3.2.1 to build the project with spark 3.2.1 dependency. It is expected that spark-3.2.1 shim layer is workable for spark-3.2.2 after our previous fixes, which needs more tests to verify.