tenzir / public-roadmap

The public roadmap of Tenzir
https://docs.tenzir.com/roadmap
4 stars 0 forks source link

Convert Snapshots into Delta Feeds #109

Open mavam opened 11 months ago

mavam commented 11 months ago

A common pattern we observe is that datasets are published and continuously updated, with the ability to tap into a feed of their changes. Rather, the original dataset simply gets overwritten. Threat intelligence feeds (like abuse.ch) are on concrete example. We would like to have the ability to turn these datasets into a feed of changes, using a delta or diff operator. This operator repeatedly fetches the entire datasets, but only yields additions, updates, or removals as output.

### Definition of Done
- [ ] Design the operator UX
- [ ] Implement
- [ ] Test with abuse.ch feeds
mavam commented 4 months ago

Now that I'm thinking about this: isn't a lot more intuitive to model this as a lookup table that is capable of emitting its diff when changes occur? Similar to lookup, this could be continuous source operator, e.g., context observe, that emits its CRUD actions as events.