Closed henridf closed 4 years ago
And in some possible future situations, such as with overlapping chunks, there simply isn't any "original order"... but that will come later.
Given that the notion of stable order of data doesn't exist in a model where data can imported in multiple batches, overlapping in the sort field, the a stable ordering will have to be defined by the system. A straightforward solution would be to use byte comparison (as we do for set ordering in zng).
Ordered merge does not have sufficient information to sort stably: when multiple records with same sort field are present at different upstreams, it does not know what their original order (*) was.
This issue consists of removing flowgraph parallelizations that rely on stable ordered merge for deterministic output.
(*) And in some possible future situations, such as with overlapping chunks, there simply isn't any "original order"... but that will come later.