anhdt1911 / scraper

Backend service for simple google search result scrapper
0 stars 0 forks source link

[Chore][Backend] Improve the scrapper code structure #3

Open malparty opened 3 months ago

malparty commented 3 months ago

Issue

The file scraper/internal/scraper/scraper.go handles many responsibilities including:

A similar issue is present in the file scraper/internal/server/server.go, which also contains code duplication between the BatchScrape and ScrapeResult functions.

Expected

To allow the application to grow while remaining clean and maintainable, the source code should be organized into sub-parts that correspond as much as possible to single responsibilities.

This also benefits testing in terms of clearly seeing "what should be tested, what should not".

anhdt1911 commented 3 months ago

Thank you for your suggestion 🙇🏼‍♂️ .

I'll will separate scraper/internal/scraper/scraper.go into smaller reusable module and also remove the duplication of BatchScrape and BatchResult