Throttling scrapper

[immangat]

Overview:

• Currently the scraper is being throttled (after around 500 request)

Tasks:

• develop a solution using proxy or user agent to bypass this limit

[immangat]

• Decreasing the time between each request to 1 seconds helps with the issue. Will be trying to decrease the time further.

[immangat]

Update:

• In-Progress
  • 4 requests per seconds, up from one. Will continue to increase the request per second to get the upper limit.