Duplicate post consolidation

[apiarian]

I would like to add a feature to the way the entries are displayed. There are some news sites (NYTimes, NPR) that have a number of RSS feeds. There are specific topic feeds and general feeds. There are sometimes articles that overlap and show up, identically, in multiple feeds. This causes those articles to show up multiple times in the feed list, even though they are essentially the same article.

I'm thinking of adding a hash to each of the entries that is based on the entry title, content, and possibly the link. These hashes would then be used to consolidate items in the aggregate lists (unread, starred, all).

Would you be interested in this feature?

[radiac]

That sounds like an interesting idea - not a problem I've run into, but can see it would be useful! The guid might be a good field to use in the hash too?

The only area that could cause problems is marking a de-duped entry as read or saved - if you just mark one, you'd probably need to mark all (limited to the user), and make sure there wasn't a read/save flag clash - but I can't think what the query code for that would look like off-hand. I presume a bit complicated - I'll leave that to you! :)

[apiarian]

Wait, what do you mean by limited to the user? The feeds are separate for the users, aren't they? Am I reading this code incorrectly? Did I miss something?

[radiac]

Yes, sorry I wasn't clear; the feeds and their entries are separate for the users, but I meant that if two users followed the same feed and had the same entries, the hashes would be the same, so any new queries designed to match by hash need to also filter by user, eg entries.objects.filter(hash=hash, feed__user=request.user). Sorry, I guess that's pretty obvious - was just thinking out loud!