AugmentedDiffStreamProcessor currently uses Spark Streaming's textFile source, which polls a directory (or bucket prefix) to populate the stream. There's no way to specify a starting point, so all JSON files that exist in the path will be loaded and processed.
We'd pushed off doing this against the interim JSON augmented diffs, but the downsides of the textFile source are becoming painful.
A side-effect of implementing a first-class source is that Row mangling can be done within the source.
...similar to
ChangesSource, etc.AugmentedDiffStreamProcessorcurrently uses Spark Streaming'stextFilesource, which polls a directory (or bucket prefix) to populate the stream. There's no way to specify a starting point, so all JSON files that exist in the path will be loaded and processed.We'd pushed off doing this against the interim JSON augmented diffs, but the downsides of the
textFilesource are becoming painful.A side-effect of implementing a first-class source is that Row mangling can be done within the source.