automch-code / google-scraping

0 stars 0 forks source link

[Chore] Improve async processing of keywords #22

Closed olivierobert closed 11 months ago

olivierobert commented 11 months ago

Issue

Upon uploading keywords, the scraping will be performed asynchronously but all keywords will be processed in a loop.

https://github.com/automch-code/google-scraping/blob/3a18502d9f4a97c31916a0e90857c9d6489412b9/backend/app/jobs/import_keywords_job.rb#L9-L21

[!WARNING] Rescueing a block with Exception is dangerous as it acts as a catch-all. It is best to raise custom errors (e.g., GoogleScraper could raise a custom error) and catch more specific errors.

Expected

The benefits is that thehe scraping of each keyword is isolated, i.e., one could error out while others could be successful.

automch-code commented 11 months ago

Hello, Olivier I have improved my async processing already, I moved the Scrapping logic to the background process after_save so each keyword will scrap in parallel without blocking each other. I am not sure my implementation is a good practice. If you have any recommendations or approaches I really happy to hear them from you kub. 😊

automch-code commented 11 months ago

this is pull request imporve async process

olivierobert commented 11 months ago

Reviewed the changes and added a few comments 📖

automch-code commented 11 months ago

hello Olivier I have checked and finished your review already. my commit moving validate logic from controller to model here I already improved the async process here

olivierobert commented 11 months ago

I have reviewed, and the improvements are good. Testing could still be improved, but the overall architecture is much more sound now.