Closed longnd closed 1 year ago
Limitations can be resolved by
saving keywords first in DB
await keywordService.saveKeywords(keywords, req.user.id);
// Enqueue keywords for background processing
keywords.forEach(async (keyword) => {
await enqueueKeywordJob(keyword);
});
We can use something like bull to enque and process jobs in background with something like :
const Queue = require('bull'); const scrapeMultipleKeywords = require('./scrapeMultipleKeywords');
// Create job queue const keywordQueue = new Queue('keywordQueue', process.env.REDIS_URL);
// Define job processing function const processKeywordJob = async (job) => { const { keyword } = job.data;
// Scrape data for the keyword const scrapedData = await scrapeMultipleKeywords([keyword]);
// Process the scraped data await keywordService.processScrapedData(scrapedData);
return scrapedData; };
Thanks for your reply. The mentioned approach should have been implemented instead of the current implementation as it is more effecient :)
Issue
The current solution to handle uploaded keyword files:
This approach has some limitations
I'm curious why didn't you pick a better option, for example: