tweag / sparkle

Haskell on Apache Spark.
BSD 3-Clause "New" or "Revised" License
447 stars 30 forks source link

mapPartitions and mapPartitionsWithIndex. #78

Closed mboes closed 7 years ago

mboes commented 7 years ago

Uses the new jvm-streaming package under the hood. Note that these functions are definable even without depending on jvm-streaming at all. But only usable if the instances defined in jvm-streaming are in scope. Since that package can only build on GHC >= 8.0.2, it means we intentionally export the same API for all compiler versions (a good thing), even if some of the API can't be used without a newer compiler.

This does not include redefining aggregate like #77 does. That can be done as part of a separate PR, if the benchmarks warrant it.

facundominguez commented 7 years ago

LGTM