Improve static datasources import & storage - Githubissues

commonsearch / cosr-back

Backend of Common Search. Analyses webpages and sends them to the index.

https://about.commonsearch.org

Apache License 2.0

122 stars 24 forks source link

Improve static datasources import & storage #26

Closed sylvinus closed 8 years ago

sylvinus commented 8 years ago

For #25:

Switch to only one RocksDB database for all the static datasources
Use protocol buffers as both storage and message format
Refactor custom import scripts into proper classes
Add minimal Wikidata datasource, first step for #10
Fix URL normalization to avoid trailing slashes for domains