Bot-detector / bot-detector-scraper

GNU General Public License v2.0
2 stars 2 forks source link

Scraper

Component Responsibility
ProxyManager the proxy manager class is responsible to get a list of proxies
Manager the manager class is responsible to initiate the workers (each worker gets one proxy), consume from the kafka player topic assign a player to a free worker to scrape, consume from the scaper topic, posting the scraped players to the api *the last responsibility should be moved to another component
Worker The worker is responsible for the scraping and its state if an error occurs the worker must return the player to the player topic
Scraper The scraper is responsible for the player object, and must gather information with the api
HighscoreApi responsible for calling the highscore api
RuneMetricsApi responsible for calling the runemetrics api