search

digitalmethodsinitiative / zeeschuimer

A browser extension to collect social media data with.
Other
159 stars 12 forks source link

Duplicates in TikTok data collection #1

Closed stijn-uva closed 1 year ago

stijn-uva commented 2 years ago

From what I can see the issue of duplicates happens with manual scrolling, and not if I use an extension to automatise the process. When present, duplicates consist of the same tiktok videos appearing 2 times, with the same metadata. In the datasets I collected (around 4.000-5.000 posts), after deleting the duplicates the datasets decreased, on average, of 1000 posts.

stijn-uva commented 1 year ago

This seems to no longer be an issue