Open jhs2jhs opened 10 years ago
wikipedia: Technical measures to stop bots http://en.wikipedia.org/wiki/Web_scraping#Technical_measures_to_stop_bots
a survey of web scraping clustering engines. https://s3-us-west-2.amazonaws.com/mlsurveys/89.pdf
http://proxy.org/ web based proxy
anonymous proxies
proxy rotation service: Such as service will have a large pool of IP addresses and rotate the IP addresses every time you make a request for a webpage, so the target website will see very few requests from the same IP address and requests from the same IP address will have a long delay between them - See more at: http://extract-web-data.com/scrape-detection-and-how-visual-web-ripper-can-help-deal-with-this-problem/#sthash.PbtlXaWa.dpuf
http://extract-web-data.com/scrape-detection-and-how-visual-web-ripper-can-help-deal-with-this-problem/