The file scraper/internal/scraper/scraper.go handles many responsibilities including:
Rotating the user agents
Fetching the Google search requests (and building the URL)
Parsing the HTML code
Mapping the data to the model
Defining the schema of the model
A similar issue is present in the file scraper/internal/server/server.go, which also contains code duplication between the BatchScrape and ScrapeResult functions.
Expected
To allow the application to grow while remaining clean and maintainable, the source code should be organized into sub-parts that correspond as much as possible to single responsibilities.
This also benefits testing in terms of clearly seeing "what should be tested, what should not".
Issue
The file
scraper/internal/scraper/scraper.go
handles many responsibilities including:A similar issue is present in the file
scraper/internal/server/server.go
, which also contains code duplication between theBatchScrape
andScrapeResult
functions.Expected
To allow the application to grow while remaining clean and maintainable, the source code should be organized into sub-parts that correspond as much as possible to single responsibilities.
This also benefits testing in terms of clearly seeing "what should be tested, what should not".