issues
search
commonsearch
/
cosr-back
Backend of Common Search. Analyses webpages and sends them to the index.
https://about.commonsearch.org
Apache License 2.0
122
stars
24
forks
source link
Refactor the indexing pipeline by adding support for plugins
#47
Closed
sylvinus
closed
8 years ago
sylvinus
commented
8 years ago
Also in this PR:
Many dependency version upgrades
Experimental use of our new
urlparse4
module for faster URL parsing
Support for new document sources beyond Common Crawl: Wikidata, URL fetch, test corpus
Update to Common Crawl's new location on S3
Also in this PR:
urlparse4
module for faster URL parsing