search

drawrowfly / amazon-product-api

Amazon Scraper. Scrape products from the amazon search result or reviews from the specific product
631 stars 181 forks source link

Add UserAgent parameter in constructor #4

Closed ShaunLWM closed 4 years ago

ShaunLWM commented 4 years ago

I realize that Amazon starts to block the scraper as early as the first 5 products (at least for me when I first started using this library). Changing the UA allows the scraper to circumvent the detection at least a little bit longer (works for more than 20+ products with 100 reviews each).

drawrowfly commented 4 years ago

I think it would be even better if we will randomize user agent for each request as i did it here https://github.com/drawrowfly/amazon-keyword-scraper-ts/blob/ce25dfc98c180cf1b2d0824e60a24311355c8b21/src/core/GrabKeywords.ts#L139

ShaunLWM commented 4 years ago

I was thinking of that as well but we are restricting it to what the user wants to use as their UA. As for my personal project, I am generating random UA from this library https://github.com/skratchdot/random-useragent. So it's up to you which direction you want this library to be headed towards.

drawrowfly commented 4 years ago

For now that will work

Thank You