Introduce conditional compilation macros for Spark-version-dependent compilation. Currently, we are relying on Spark-version-specific source directories and they work nicely, but sometimes it's not easy to extract differences into separate files without duplicating a lot of code, introducing performance overhead, and/or impacting the code readability.
For example, in #443, I added annotations for StructType serialization/deserialization because without them it doesn't work in Spark 2.4. However, those annotations do unnecessary extra things for Spark 3+. There are ways to handle the issue with version-specific source directories, but using macros is the cleanest: no duplication, no runtime overhead, and readable.
clee704
commented
3 years ago
What is the context for this pull request?
What changes were proposed in this pull request?
Introduce conditional compilation macros for Spark-version-dependent compilation. Currently, we are relying on Spark-version-specific source directories and they work nicely, but sometimes it's not easy to extract differences into separate files without duplicating a lot of code, introducing performance overhead, and/or impacting the code readability.
For example, in #443, I added annotations for
StructTypeserialization/deserialization because without them it doesn't work in Spark 2.4. However, those annotations do unnecessary extra things for Spark 3+. There are ways to handle the issue with version-specific source directories, but using macros is the cleanest: no duplication, no runtime overhead, and readable.Does this PR introduce any user-facing change?
No
How was this patch tested?
Unit test (see MacroTest.scala)