Deduplicate social media sharing events

citp / news-disinformation-study

A research project on how web users consume, are exposed to, and share news online.

8 stars 2 forks source link

Closed jonathanmayer closed 4 years ago

jonathanmayer commented 4 years ago

A couple cases we need to handle:

The user includes the same URL multiple times in a post. We should probably record this as one share.
An API POST includes the same URL in both the post body and as a post attachment (e.g., an enrichment). We should definitely record this as one share.

When doing deduplication, we should normalize URLs (e.g., stripping out URL parameters).

akohlbre commented 4 years ago

a65ec21