Create a CrawlToolUtils class with helper methods to set up page,sitemap and robots fetchers and the url lengthener.

ScaleUnlimited / flink-crawler

Continuous scalable web crawler built on top of Flink and crawler-commons

Apache License 2.0

51 stars 18 forks source link

Create a CrawlToolUtils class with helper methods to set up page,sitemap and robots fetchers and the url lengthener. #142

Closed vmagotra closed 6 years ago

vmagotra commented 6 years ago

Currently the CrawlTool has private methods to set up the url lengthener, and the page,sitemap and robots fetchers. Move these methods into a utility class so that tools that are built on top of flink-crawler can make use of them.