neilconway
commented
11 years ago Related: the rescan logic for the grouping operator is also overly conservative. Group is marked as always-rescan. However, if no new input to the grouping operator arrives, there is no need to rescan it or blowaway downstream state. Similarly, if new input arrives but it only belongs to new groups, there is no need to do a full rescan.
For both notin and group, adjusting the rescan behavior based on the actual runtime dataflow will take some work, however. Also note that the current rescan logic probably assumes that NM operators unconditionally rescan in some places (e.g., when we see a deletion to a downstream operator and want to force its inputs to be rescanned, we only consider elements in the same strata; if we had to explicitly cause rescan of NM operators, not sure offhand if this would still be sufficient).
Right now, notin is marked for rescan at every tick. This is too conservative: if the "left" operand to the notin (i.e., the input against which the notin has a positive dependency) only grows, the output of the notin will increase monotonically, and hence we don't need to invalidate/recompute downstream state.