search

ramyasaimullapudi / ItemStockTracker

MIT License
0 stars 3 forks source link

Multithreading scrapers to fetch stock availability parallely #22

Open krishnasaurabh opened 3 years ago

krishnasaurabh commented 3 years ago

To fetch data parallely we can use a thread pool to scrape data.

krishnasaurabh commented 3 years ago

I have implemented the feature but i am hitting the captcha problem more frequently now, and everything shows "No stock info" for amazon links.

Amazon seems to block the request if there is less than 2 seconds between requests. I have tried to see if rotating proxies will work, but the list of proxies available on the internet are all either expired or dont work, we need to purchase something to bypass the captcha.

Should we parallelize the bestbuy links and process the amazon links in sequential order?

@LakshmiSwethaGavini @arjunptm @rhnprabhune Please throw your suggestions.