Small nuisance using spores with Spark

This works:

      val wc2a = input
        .flatMap(line => line.split("regex"))
        .countByValue() // Returns a Map[T, Long]

But when we add a spore wrapping the expression in the flatMap it doesn't:

      val wc2a = input
        .flatMap(spore {
          val regex = alphabeticRegex
          line => line.split(regex)
        })
        .countByValue() // Returns a Map[T, Long]

Error:

[error] /data/rw/code/scala/spores-spark-tutorial/src/main/scala/sparktutorial/WordCount3.scala:68: type mismatch;
[error]  found   : anonspore$macro$17
[error]  required: String => TraversableOnce[?]
[error]         .flatMap(spore {
[error]                        ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 3 s, completed Dec 28, 2016 2:33:44 PM

This error happens because when we add the spores abstraction, the Scala compiler cannot reconcile the return type parameter of the spore (which is fixed to be an Array[String]) with the one that the flatMap operation expects. This means that it cannot resolve the implicit from Array to Traversable and it has to be done manually with:

      val wc2a = input
        .flatMap(spore {
          val regex = alphabeticRegex
          line => line.split(regex).toTraversable // see the .toTraversable call
        })
        .countByValue() // Returns a Map[T, Long]

I don't know the exact reason of this failure, but it seems related to a special handling that the typechecker does when it finds lambdas. When we use a spore, this special handling is not executed anymore and the implicit resolution to convert from Array to Traversable doesn't happen.

scalacenter / spores

Small nuisance using spores with Spark #26