backend + research: improve scraper performance (reduce time)

arugyani / shopIQ

LLM powered high-consideration research tool for UTD senior project

1 stars 0 forks source link

backend + research: improve scraper performance (reduce time) #97

Open divyamk opened 2 months ago

divyamk commented 2 months ago

Issue discovered: non google shopping links gets scraped and the scraper takes 30 seconds to hang and move on. Non google shopping links should be removed.

divyamk commented 2 months ago

May be able to use puppeteer clusters to make faster.