search

NICTA / scoobi

A Scala productivity framework for Hadoop.
http://nicta.github.com/scoobi/
482 stars 97 forks source link

Make the inputCheck on Scoobi DataSources a modifiable parameter #258

Closed etorreborre closed 11 years ago

etorreborre commented 11 years ago

Something like:

TextInput.fromTextFile("path", check = (p: Path) => p.size > 0)
etorreborre commented 11 years ago

I changed the initial proposal so that the published version is:

val list = fromTextFile(Seq("missing"), check = Source.noInputCheck)
// or
val list = fromTextFile(Seq("missing"), check = (paths: Seq[Path], sc: ScoobiConfiguration) => ???)

The ScoobiConfiguration object is passed in order to access the FileSystem object:

sc.fileSystem