search

wchan757 / similarweb_scraper

Scraping similarweb with pywinauto
MIT License
22 stars 4 forks source link

Cannot get it to work #1

Open kilobyte2007 opened 4 years ago

kilobyte2007 commented 4 years ago

Does this still work of now? I am getting the 520 response code while making the scrape request. The url prefix is right. Is it something related to their anti-bot mechanisms?

wchan757 commented 4 years ago

Hi kilobyte2007, Sorry for my late reply.

The method of proxycrawl api doesn't work anymore. As a result, I figure out a desperate approach: Using the pywinauto to download the html file first and scrape all the data locally.

It work so far. I have uploaded all the code in this repositories. you can try it

Thank you very much

adnantamimi commented 1 year ago

@wchan757 Hey, I have been trying the steps you mentioned to download the html and then scrape it but I keep running into this error. ElementNotFoundError: {'title_re': '.*Market Share Stats', 'backend': 'uia', 'visible_only': False}