search

Deepakchawla / Mobile-Phone-Dataset-GSMArena

Python script for creating Mobile Phones Dataset on GSMArena website.
MIT License
59 stars 46 forks source link

Too Many Requests (response code 429) #8

Closed AwalDeep closed 3 years ago

AwalDeep commented 3 years ago

IP is blocked by the website after few hits. How to spoof the ip or solve this issue?

krm1337 commented 3 years ago

same here, Too Many Requests. Not like the script doesn't throwing errors anymore though: current error: PS D:\DOWNLOADS\Mobile-Phone-Dataset-GSMArena-master> py .\gsmarena_scraping.py Traceback (most recent call last): File "D:\DOWNLOADS\Mobile-Phone-Dataset-GSMArena-master\gsmarena_scraping.py", line 163, in obj.save_specification_to_file() File "D:\DOWNLOADS\Mobile-Phone-Dataset-GSMArena-master\gsmarena_scraping.py", line 129, in save_specification_to_file phone_brand = self.crawl_phone_brands() File "D:\DOWNLOADS\Mobile-Phone-Dataset-GSMArena-master\gsmarena_scraping.py", line 46, in crawl_phone_brands table = soup.find_all('table')[0] IndexError: list index out of range

@AwalDeep Could you make it run at the end ?

AwalDeep commented 3 years ago

@krm1337 The error you are getting is because of the same issue I had mentioned above. You're getting empty list because IP is blocked and there is no data scraped.

Deepakchawla commented 3 years ago

code has been updated now guys will not get the issue of the too many requests.