Closed olivierobert closed 11 months ago
Hello, Olivier I have improved my async processing already, I moved the Scrapping logic to the background process after_save so each keyword will scrap in parallel without blocking each other. I am not sure my implementation is a good practice. If you have any recommendations or approaches I really happy to hear them from you kub. 😊
Reviewed the changes and added a few comments 📖
I have reviewed, and the improvements are good. Testing could still be improved, but the overall architecture is much more sound now.
Issue
Upon uploading keywords, the scraping will be performed asynchronously but all keywords will be processed in a loop.
https://github.com/automch-code/google-scraping/blob/3a18502d9f4a97c31916a0e90857c9d6489412b9/backend/app/jobs/import_keywords_job.rb#L9-L21
Expected
The benefits is that thehe scraping of each keyword is isolated, i.e., one could error out while others could be successful.