search

twitter / summingbird

Streaming MapReduce with Scalding and Storm
https://twitter.com/summingbird
Apache License 2.0
2.14k stars 267 forks source link

Removing internal config setup from scalding platform #629

Closed ianoc closed 8 years ago

ianoc commented 8 years ago

1) Both Timestamp and BatchID become any val classes to erase them at runtime ---> Knock on effect is to have them not extend ordered. Provide an implicit enrichment to add those methods back using the ordering on the companion object 2) Remove the internal config handling from the scalding platform, delegating to the Scalding Config class instead.

reconditesea commented 8 years ago

uh, this hurts :( But yeah, I think we should just remove this file. Even I cannot quite remember why I/we want to add this DirectoryBatched store in the first place.

johnynek commented 8 years ago

Looks good to me. I'm slightly worried about the source incompatibility here, but the point is better control over how the job is run and that is really important and not a small issue.

ianoc commented 8 years ago

Awesome, many thanks for the review help!