issues
search
CodeForBc
/
airbnb-regulation
MIT License
3
stars
2
forks
source link
Throttling scrapper
#6
Open
immangat
opened
3 months ago
immangat
commented
3 months ago
Overview:
Currently the scraper is being throttled (after around 500 request)
Tasks:
develop a solution using proxy or user agent to bypass this limit
immangat
commented
3 months ago
Decreasing the time between each request to 1 seconds helps with the issue. Will be trying to decrease the time further.
immangat
commented
3 months ago
Update:
In-Progress
4 requests per seconds, up from one. Will continue to increase the request per second to get the upper limit.
Overview:
Tasks: