twitter / summingbird

Streaming MapReduce with Scalding and Storm
https://twitter.com/summingbird
Apache License 2.0
2.14k stars 267 forks source link

Use scalding.Execution in scalding platform instead of custom monad #710

Open johnynek opened 7 years ago

johnynek commented 7 years ago

The scalding platform uses a custom planning monad stack which has reader-like behavior to read the configuration and mode and state-like behavior to update the view of daterange we can compute.

This code is fairly hard for a new person to start hacking on since it is in parallel computing two things: how to compute data and what time range we can compute.

The scalding.Execution monad came along after this code and is tested outside of summingbird. It does not have the state-like features, but it can handle the reader-like aspect of configuration and mode, and it does give access to changing configuration for parts of the job.

Related to #136