search

tweag / sparkle

Haskell on Apache Spark.
BSD 3-Clause "New" or "Revised" License
447 stars 30 forks source link

Possibility of Migrating to Spark Connect #177

Open MrPowers opened 2 months ago

MrPowers commented 2 months ago

Newer language clients are being built with Spark Connect, see Spark Connect Rust for example.

Spark Connect should make it so language bindings are easier to maintain and more fully featured. Would you be willing to consider making sparkle work with Spark Connect?

facundominguez commented 2 months ago

Hello @MrPowers. We could consider merging contributions related to Spark Connect. It is unclear to me what support is needed on the sparkle side, but feel free to open issues to discuss the use cases and the technical obstacles.

MrPowers commented 2 months ago

@facundominguez - Spark Connect is a new architecture that decouples the client from the server. This would allow for Haskell users to run Spark computations without any Java/JVM dependencies on the client side. This page has a good overview.

Decoupling the Haskell language bindings and freeing this implementation from the JVM should provide a better experience for the Haskell community. When the server JVM/Java version upgrades, the client code shouldn't have to upgrade. That should also make the codebase more maintainable.

Looking forward to collaborate on this and thanks for making such a cool library.