commonsearch / cosr-back

Backend of Common Search. Analyses webpages and sends them to the index.
https://about.commonsearch.org
Apache License 2.0
122 stars 24 forks source link

Improve datasources import #25

Open sylvinus opened 8 years ago

sylvinus commented 8 years ago

We currently have ad-hoc scripts in https://github.com/commonsearch/cosr-back/tree/master/scripts to import our various datasources, each of them creating a different rocksdb database.

This is not ideal and should be improved in the following ways: