search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

Add a spark backend #1741

Open johnynek opened 6 years ago

johnynek commented 6 years ago

To test the modularity, a basic spark backend based on RDDs should be implemented.

Should be easy to do to based on the cascading backend and the memory backend.

EncodePanda commented 6 years ago

Can you point to the implementation of the inmem backend?

johnynek commented 6 years ago

https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/memory_backend/MemoryBackend.scala

@rabbitonweb

sorry for the latency. Missed this somehow.

johnynek commented 6 years ago

see #1782

That gets the memory backend usable (I think).

I'm going to work on a basic spark implementation soon (next couple of weeks).

johnynek commented 6 years ago

this has begun. See #1832