[Question] What is the plan to work around the limitations of mass-searching keywords

[longnd]

The scrapper does not implement any mechanism to work around the limitations of mass-searching keywords. If the user upload a long list of keywords (100 keywords for example), it is likely that Google will block those continuous requests.

Could you share your plan to overcome that challenge?

[Vivek-sudo]

To overcome the challenge of potential Google blocking due to continuous requests, we can implement a few strategies:

• Rate Limiting and Delay: Implement rate limiting by introducing a delay between each request. Google can detect rapid-fire requests, so introducing a delay (e.g., a few seconds) between each keyword search can help reduce the chances of being detected as a bot.
• Using Proxies: Rotate through a pool of IP proxies. Each request will be made using a different proxy, making it harder for Google to identify and block your IP address.
• Session Persistence: Maintain a persistent browser session (using something like puppeteer-cluster), which can help avoid triggering bot detection since a single browser session will be making requests over time.