[Question] Why not using background job for the scrapping process?

[longnd]

Issue

The current solution to handle uploaded keyword files:

• parse the file
• loop through the keyword list
  • using Puppetee to perform the search
  • extract search result
• Insert the keywords with their scrapping result to the DB

This approach has some limitations

• If the scrapping process for one keyword is broken for any reasons, the entire list of keyword will not be saved into DB, all completed scrapping results will be gone.
• The user will need to wait until the request is completed on the server side to search the research of there uploaded keyword, there is no way for them to search the research of each keyword, even it was completed. If the keyword list is long, it can be timed out.

I'm curious why didn't you pick a better option, for example:

• parse and save the keywords into the DB first
• enqueue each keyword as a background job
• a worker to pick up the job to process

[Vivek-sudo]

Limitations can be resolved by

• saving keywords first in DB

       await keywordService.saveKeywords(keywords, req.user.id);
  
          // Enqueue keywords for background processing
          keywords.forEach(async (keyword) => {
              await enqueueKeywordJob(keyword);
          });

• We can use something like bull to enque and process jobs in background with something like :

  const Queue = require('bull'); const scrapeMultipleKeywords = require('./scrapeMultipleKeywords');

  // Create job queue const keywordQueue = new Queue('keywordQueue', process.env.REDIS_URL);

  // Define job processing function const processKeywordJob = async (job) => { const { keyword } = job.data;

  // Scrape data for the keyword const scrapedData = await scrapeMultipleKeywords([keyword]);

  // Process the scraped data await keywordService.processScrapedData(scrapedData);

  return scrapedData; };

[longnd]

Thanks for your reply. The mentioned approach should have been implemented instead of the current implementation as it is more effecient :)