The amazon scraper sometimes gets blocked and retrieves just very few products. I suggest adding some randomization in which order the categories of a merchant are accessed so that every time we scrape we start with different categories. This would help to increase the number of unique products, regardless of being blocked at some point.
In addition, we could also add some randomization to the order in which we access the products of a SERP page. Maybe both of these suggestions would also help to decrease the chance of being detected as a bot.
The amazon scraper sometimes gets blocked and retrieves just very few products. I suggest adding some randomization in which order the categories of a merchant are accessed so that every time we scrape we start with different categories. This would help to increase the number of unique products, regardless of being blocked at some point.
In addition, we could also add some randomization to the order in which we access the products of a
SERPpage. Maybe both of these suggestions would also help to decrease the chance of being detected as a bot.