Refactoring: Move WARC record outside hadoop module

dkpro / dkpro-c4corpus

DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate removal, language detection, and near-duplicate removal.

https://dkpro.github.io/dkpro-c4corpus

Apache License 2.0

50 stars 8 forks source link

Refactoring: Move WARC record outside hadoop module #11

Closed habernal closed 8 years ago

habernal commented 8 years ago

Reading WARC files is independent of Hadoop, can be used also for local files