search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.48k stars 703 forks source link

Beam-Dataflow support for Scalding #1940

Closed Franco1010 closed 2 years ago

Franco1010 commented 2 years ago

This is a first PR for beam support to scalding

It now extends support to basic operations in SimpleWrite only:

Added support for reading from Source path and Iterable Added support for writing into Sink path Added KryoCoder sharing kryo instances and buffers Added unit tests for BeamMode end to end

CLAassistant commented 2 years ago

CLA assistant check
All committers have signed the CLA.

johnynek commented 2 years ago

Looks like the microsite test failed:

error: File /home/travis/build/twitter/scalding/docs/docs does not exist.

Exception: sbt.TrapExitSecurityException thrown from the UncaughtExceptionHandler in thread "run-main-0"

[error] Nonzero exit code: 1

[error] (docs / Compile / runMain) Nonzero exit code: 1

[error] Total time: 168 s (02:48), completed Aug 18, 2021 2:58:18 AM

The command "./sbt ++$TRAVIS_SCALA_VERSION clean docs/makeMicrosite" exited with 1.
ttim commented 2 years ago

@johnynek it's independent failure and was failing since I upgraded sbt (and fails on develop too =( ). I'll tackle it this week.