TechAndCheck / tech-and-check-alerts

Daily tip sheet for fact checkers
MIT License
13 stars 6 forks source link

What happens if the same account is in two lists? #352

Open slifty opened 4 years ago

slifty commented 4 years ago

When we scrape, we have some logic to prevent redundant scrapes of the same claim.

If an account is listed in two separate newsletters, does this mean a given claim will only appear in one of them, or is it smart enough to handle that.

reefdog commented 4 years ago

It's "smart" enough to handle that, in that claims are not fetched by list name. The newsletter fetches the accounts associated with the newsletter's list, and then fetches claims associated with those account names. (The claims themselves are associated with the account directly, not the account-list tuple.)

reefdog commented 4 years ago

We aren't doing any de-duping of the account names themselves when we scrape, but I believe that's handled by the scrape URL history that you allude to.