The 2 current webscrapers are the same. I assume that future webscrapers will follow a similar pattern.
That being said, I plan to create an abstracted version of the current webscrapers that take a config variable that contain specifics for each webscraper. I.e. a walmart config, an amazon config, and a target config variable.
We can add an additional parameter that determines which scrapers to run, default being all. With this parameter, we can easily only call the specific configs to run the scraper on.
TODO:
[ ] Determine what's different between current scrapers
[ ] Determine config structures
[ ] Begin work on Target and possibly other web scrapers
The 2 current webscrapers are the same. I assume that future webscrapers will follow a similar pattern.
That being said, I plan to create an abstracted version of the current webscrapers that take a config variable that contain specifics for each webscraper. I.e. a walmart config, an amazon config, and a target config variable.
We can add an additional parameter that determines which scrapers to run, default being all. With this parameter, we can easily only call the specific configs to run the scraper on.
TODO: