search

ScaleUnlimited / flink-crawler

Continuous scalable web crawler built on top of Flink and crawler-commons
Apache License 2.0
51 stars 18 forks source link

Get rid of non-focused crawl support in code #153

Closed kkrugler closed 6 years ago

kkrugler commented 6 years ago

We should treat every crawl as a focused crawl, so we don't need a FocusedPageParser. And we should always provide a page scorer, with a default.