search

rtp-atw / nimble-interview

Nimble Repository
https://nimble-interview.vercel.app
MIT License
0 stars 0 forks source link

[Feature] Implement validation logic to process less than 100 keywords #21

Open olivierobert opened 8 months ago

olivierobert commented 8 months ago

Issue

The current implementation does not prevent the processing of more than 100 keywords:

https://github.com/rtp-atw/nimble-interview/blob/d49c56f1eb5ebf5a2ed6aa4036044b29c250c275/backend/pkg/keywords/upload.go#L38-L40

However it is a requirement:

requirement

Expected

To improve the UX, the content of the CSV should be processed and validated before starting any scraping. The UI must display an error if a user uploads more than 100 keywords.

[!Note] I also wonder what is the purpose of the end variable in this loop. Is it to process keywords in batches of 5 🤔 I do not quite get the intent

rtp-atw commented 8 months ago

Answer for end variable and the loop, I'm planning to refactor this process.

The main idea is to limit each queue consumer to process only up to five keywords at a time. Previously, all processes, like generating reports and scraping, were handled by the consumer.

Now, I've changed it so that the report is generated first. After that, the consumer's role is solely to update the results in the report.

olivierobert commented 8 months ago

I have added a comment to the already-merged PR.