Duplicate activities when scraper is ran at different times causing the scraping time range to overlap

ohcnetwork / leaderboard

Leaderboard collects data from GitHub and Slack to show off the work of our open source contributors

https://contributors.ohc.network/

MIT License

18 stars 74 forks source link

Duplicate activities when scraper is ran at different times causing the scraping time range to overlap #507

Closed rithviknishad closed 2 months ago

rithviknishad commented 2 months ago

When the script that ran on the night of 22nd Aug. to fill missing data, and the scheduled scraper CRON job ran on the morning of 23rd aug., the default time range of the scraper caused to yield duplicate activities.

Action Items

[x] Ensure scraper always performs de-duplication of activities of user data before the data is saved and commited.