It's a big one; sorry about that. In the process, I pushed quite a bit of code from pigpen.cascading and the java UDFs into pigpen.cascading.runtime. This probably churned the code a little more than I would have liked, but I think it ended up in a good place.
Here's the overview of the cascading changes:
The command->flowdef fn is passed the command definition, a sequence of its ancestors (tuples of pipes and commands), and the flowdef
The new command->flowdef+ manages pipes produced by commands and looking up ancestors for the next command
Instead of passing around a map of taps, sinks, pipes, etc, each command returns a Pipe
Taps & Sinks are added directly to the flowdef. (I'd like to revisit this, but I couldn't get the old way working)
Projections are evaluated entirely on the runtime side; we don't need to figure out field mappings ahead of time
:reduce replaces :group-all and is now a separate command
I added oven rewrites to merge any group/fold or reduce/fold combinations. In cascading, these are one operation, so we make them one operation in the model
:generate is dependent on the previous command
There's a new :noop command for working around self-joins
@pkozikow
It's a big one; sorry about that. In the process, I pushed quite a bit of code from
pigpen.cascadingand the java UDFs intopigpen.cascading.runtime. This probably churned the code a little more than I would have liked, but I think it ended up in a good place.Here's the overview of the cascading changes:
command->flowdeffn is passed the command definition, a sequence of its ancestors (tuples of pipes and commands), and the flowdefcommand->flowdef+manages pipes produced by commands and looking up ancestors for the next commandPipe:reducereplaces:group-alland is now a separate command:generateis dependent on the previous command:noopcommand for working around self-joins