Refactored the shim layer code for spark 3.2.x. A common module for spark 3.2.x shims is introduced to maintain common code for spark321 and spark322. The spark322 shim layer module was added and some classes applicable for spark322 are also kept in this module, like what we did for other spark versions.
We have spark-3.2.1 and spark-3.2.2 profile added in the PR. To compile for spark321 or spark322, use -Pspark-3.2.1 and -Pspark-3.2.2 respectively in maven building. And -Pspark-3.2 is still kept, which uses spark321 as default.
github-actions[bot]
commented
1 year ago
What changes were proposed in this pull request?
Refactored the shim layer code for spark 3.2.x. A common module for spark 3.2.x shims is introduced to maintain common code for spark321 and spark322. The spark322 shim layer module was added and some classes applicable for spark322 are also kept in this module, like what we did for other spark versions.
We have
spark-3.2.1andspark-3.2.2profile added in the PR. To compile for spark321 or spark322, use-Pspark-3.2.1and-Pspark-3.2.2respectively in maven building. And-Pspark-3.2is still kept, which uses spark321 as default.How was this patch tested?
TPC-H/DS.