search

zackthoutt / wine-deep-learning

Exploring applications of deep learning to the world of wine
120 stars 62 forks source link

Bypass anti-scraping #1

Closed ryanxjhan closed 4 years ago

ryanxjhan commented 4 years ago

It looks like that the site has implemented some kind of anti-scraping mechanism. The site will ban me from accessing it for about 100 reviews scraped. Is there a solution to this?

manycoding commented 4 years ago

You could limit the number of concurrent requests, default it's 10. 10 works for me without any ban.

zackthoutt commented 4 years ago

You can also try scraping from a different IP. They (understandably) may have more sophisticated anti-scraping software now that black lists IP addresses if you hammer the site too hard.