PoC of raw scalacheck propery for Spark Streaming

[juanrh]

• Ensure test cases are respected as dstreams, in the sense that batch are generated consecutively as they appear in the test case.
  • having some empty batches due to sync problems is not acceptable, even if empty batches are ignored, because that makes window operations unusable DONE
  • empty batches due to sync or buffering problems lead to mixing different batches of the test case in the same batch executed by Spark. This is not acceptable and there should be a test case for this
• support empty batches with Either: currently we cannot create an empty batch because we emit nothing for empty batches. So replace multiplexing DStream[(TestId, A)] by DStream[(TestId, Either[A])] where Left is used for empty batches, and Right for non empty batches. We maintain an invariant that in the multiplexing DStream for each key we either have one or more Right, or a single Left DONE, Option is more suitable because there is no record associated to empty test cases

[juanrh]

See https://gist.github.com/juanrh/c719ca82f441f8dacd59#file-testcasedictinputdstream-scala for preliminary version of custom InputDStream that allows dynamic queueing of DStreams. This exploits the fact that the InputDStream is running at the driver, in the style of the Kafka direct API. On the other hand, note this design is quite bad as it mixes the receiver with the test case manipulation, which result in unnecessary coupling, and a not very reusable InputDStream. Also tracking the origin of the test cases leads to using dictionaries, instead of simpler List, which are more efficient for the full scans that we have to perform here

[juanrh]

First PoC respecting batch limits stored as a lightweight tag at https://github.com/juanrh/sscheck/releases/tag/directStreamingBatchesOkPoC