Optimize scraping performance

mkalam-alami / ludumdare-feedback-friends

A (defunct) incentive to comment on games, initially made for LD36

http://feedback.ld.intricati.com

MIT License

12 stars 5 forks source link

Open mkalam-alami opened 7 years ago

mkalam-alami commented 7 years ago

Right now the scraping loop takes 2 hours.

[x] Fetching all the author names from the DB before scraping should help bypass the 3rd HTTP call (= to author nodes) most of the time.
[x] Tune down the HTTP throttling, it will remain around 1 req/s anyway
[ ] A refactoring could let us make batch entry updates with the same HTTP call

UPDATE: Now down to an estimated ~80mn