search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.49k stars 703 forks source link

Spark backend #1832

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

This is a very basic beginning to a spark backend.

It is not complete, but does support map-only operations.

There is one big question: can we really just lie to spark and say we have AnyRef everywhere? I think it may just make serialization worse (kryo writing the classnames), but maybe we can circumvent that later since scalding allows configs to have registered classes named, maybe we can pass that information to spark somehow.

cc @ianoc

johnynek commented 6 years ago

@ianoc can you take a look?

This is not finished, but what we have is testable and in the interest of keeping the PRs small, I'd like to merge this and then follow up with more parts:

  1. (this) basic framework, map-only operation support.
  2. reduce operation support.
  3. join support
  4. full Mode/Execution support
johnynek commented 6 years ago

build fails because spark is not there for 2.12. Will remove spark 2.12 from the CI.

ianoc commented 6 years ago

Some comments but they are more of an ongoing discussion than anything else. LGTM

johnynek commented 6 years ago

thanks for the comments. these seem like good points to keep in mind as we optimize. All of your comments are doable for sure. Will address in the follow ups.