aaronpk / Aperture

Aperture is a Microsub server. Currently in beta.
https://aperture.p3k.io
Apache License 2.0
69 stars 5 forks source link

Add option to deduplicate entries within a channel across sources #40

Open aaronpk opened 6 years ago

aaronpk commented 6 years ago

Currently each source is treated independently, and if two sources end up having the same entry, the entry will appear twice in the channel. This is only a problem when for example I add a Twitter search from granary.io as a source, as well as have my streaming search script running, and a tweet matches both.

A channel should have an option to dedupe entries by the entry URL across all sources in the channel. This is potentially unsafe, since any source can claim an entry URL on any domain, so this should not be the default.