NICTA / scoobi

A Scala productivity framework for Hadoop.
http://nicta.github.com/scoobi/
482 stars 97 forks source link

combine() line from word count example will not compile #276

Closed jastice closed 11 years ago

jastice commented 11 years ago

I copied the word count example from http://nicta.github.io/scoobi/guide-SNAPSHOT/guide/com.nicta.scoobi.guide.QuickStart.html#Write+your+code almost verbatim, but the line .combine((a: Int, b: Int) => a + b) will not compile, giving the error message type mismatch; found : (Int, Int) => Int required: com.nicta.scoobi.core.Reduction[?]

This works for me however:

val combiner: Reduction[Int] = new Reduction[Int] { val reduce = (a:Int, b:Int) => a+b }
val counts = lines.flatMap(_.split("\\s"))
                  .map(word => (word, 1))
                  .groupByKey
                  .combine(combiner)
etorreborre commented 11 years ago

Sorry about that. I've migrated a few examples in the documentation so that they can be compiled but not this one.

An easier fix is to use the Reduction.Sum.int function directly:

import Reduction._
val counts = lines.flatMap(_.split("\\s"))
                          .map(word => (word, 1))
                          .groupByKey
                          .combine(Sum.int)

There are lots of other functions and combinators to build Reduction objects when you need them. For example if you need to create a Reduction which will work on a pair and do:

  val red: Reduction[(Int, Int)] = Sum.int *** maximum // or Sum.int zip maximum
jastice commented 11 years ago

Okay, thanks for the extra info!