Closed ddebrunner closed 4 years ago
Any reason why an operator rather than a function?
I hadn't thought about an function, but that would require the flattening to be done by every caller in a Custom operator.
The operator would submit N tuples for each input tuple, where N is the size of the array in the JSON object represented by the input tuple.
I just realized you can't do this with a java function, so now I think an operator is the best approach here.
Some data sources provide a single JSON object containing an array with multiple values that might be better handled as individual tuples, e.g. SFPark data returns a JSONObject containing some meta data and then an array of parking locations.
Something like:
steam<Json> F = FlattenJSONArray(I) { params array: "locations"; // attribute to be flattened metadata: "timestamp", "success"; // optional list of meta-attributes to copy into each tuple }