search

hartleybrody / public-amazon-crawler

676 stars 227 forks source link

Which Settings recommended for proxies? #12

Open Marow opened 6 years ago

Marow commented 6 years ago

Hi, i tried some proxies cause that much requests costs very much with the time.

So i tried like that site http://pubproxy.com/#premium but which settings are needed for this crawler? I have proxybonanze already, working fine with SOCKS5, so i tried so much on pubproxy but its not working...

rochos-foniem commented 5 years ago

I tried pubproxy but their proxies are blocked and their system is not compatible with the script, then I tried proxycrawl and it worked perfectly with the crawler, I used their API so you just have to replace amazon urls with the api url and you can continue using the script to crawl amazon

rochenka commented 5 years ago

yes I got similar issues and solved with this service https://proxycrawl.com/scraping-api-avoid-captchas-blocks I also used the proxycrawl python package.