edsu / feediverse

Send RSS/Atom feeds to Mastodon
MIT License
116 stars 25 forks source link

Add de-duping to feediverse #46

Closed astyfoo closed 5 months ago

astyfoo commented 1 year ago

Setting up some bots for scraping RSS and pushing to Mastodon recently, I notice many larger sites (e.g. CNBC) update their entries either to make updates to make changes, or just push old entries to the top. This causes feediverse to re-toot entries, sometimes several times in a row.

Added a command line flag (by default disabled) to add some basic de-duplication and maintain its state in the .feediverse file.

astyfoo commented 1 year ago

Hello, first of all happy holidays!

Second, I'm hoping this can be merged with upstream - if there is any additional documentation or refactoring needed please just say the word.

edsu commented 5 months ago

Sorry it took so long to get to this!