Support for multiple Spark versions

salesforce / TransmogrifAI

TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning

https://transmogrif.ai

BSD 3-Clause "New" or "Revised" License

2.24k stars 393 forks source link

Support for multiple Spark versions #462

Open gerashegalov opened 4 years ago

gerashegalov commented 4 years ago

Problem We need to be able to support multiple Spark versions to avoid painful upgrades

Solution

Create build profiles capturing dependencies of a particular Spark version. Add a Shim library with the stable facade translating API's that change let us say from 2.3.2 to 2.4. x to 3.x

Alternatives continue being in lock step with Spark and suffer

Additional context Support for wider variety of users with differing pace of Spark version adoption

koertkuipers commented 4 years ago

not just multiple spark versions... also multiple scala versions

SemanticBeeng commented 4 years ago

Maybe we can reuse the approach from frameless https://github.com/typelevel/frameless

Frameless intentionally does not have a compile dependency on Spark. This essentially allows you to use any version of Frameless with any version of Spark.

Since the 0.9.x release, Frameless is compiled only against Scala 2.12.x.