search

hudsonbay / google_scraper_live_view

Application for extracting large amounts of data from the Google search results page
16 stars 0 forks source link

Synchronous Keyword Processing #2

Closed olivierobert closed 2 years ago

olivierobert commented 2 years ago

GoogleScraper does a good job at externalizing the core business logic for the scrapping work outside of the LiveView 👏 The keywords are processed via a loop synchronously.

https://github.com/hudsonbay/google_scraper_live_view/blob/22ee211c8df4d802fe0879824e005b05ebb4aa95/lib/google_scraper.ex?_pjax=%23js-repo-pjax-container%3Afirst-of-type%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%3Afirst-of-type%2C%20%5Bdata-pjax-container%5D%3Afirst-of-type#L14-L24

Say the end-user does upload 100 keywords, this would take a while, especially considering the added random sleep 🙈

The tricky part of the code challenge lies more in the actual processing of keywords. The latter should be processed asynchronously allowing to display the results back to the user as soon as possible.

hudsonbay commented 2 years ago

Yes, you are right. If the user uploads 100 keywords it will be like forever 😄. Even with a loading spinner it's annoying. That can be solved. Let's give the users a better experience. I'll work on that and insert it on the database as soon as the keyword is found