Open mavam opened 11 months ago
Now that I'm thinking about this: isn't a lot more intuitive to model this as a lookup table that is capable of emitting its diff when changes occur? Similar to lookup
, this could be continuous source operator, e.g., context observe
, that emits its CRUD actions as events.
A common pattern we observe is that datasets are published and continuously updated, with the ability to tap into a feed of their changes. Rather, the original dataset simply gets overwritten. Threat intelligence feeds (like abuse.ch) are on concrete example. We would like to have the ability to turn these datasets into a feed of changes, using a
delta
ordiff
operator. This operator repeatedly fetches the entire datasets, but only yields additions, updates, or removals as output.