Refactor the indexing pipeline by adding support for plugins

commonsearch / cosr-back

Backend of Common Search. Analyses webpages and sends them to the index.

https://about.commonsearch.org

Apache License 2.0

122 stars 24 forks source link

Closed sylvinus closed 8 years ago

sylvinus commented 8 years ago

Also in this PR:

Many dependency version upgrades
Experimental use of our new urlparse4 module for faster URL parsing
Support for new document sources beyond Common Crawl: Wikidata, URL fetch, test corpus
Update to Common Crawl's new location on S3