search

NCBI-Hackathons / seqr

Creative Commons Zero v1.0 Universal
12 stars 2 forks source link

Store metadata outside of Solr #34

Open averagehat opened 8 years ago

averagehat commented 8 years ago

We know that storing metadata within solr makes the search slower. We could add a second embedded database to do the metadata look-up using solr's search results. Would it make sense to have a second index for the metadata using solr rather than introducing a second database technology?

For the server-based solution, maybe a second server would be appropriate.

lianyi commented 8 years ago

Yes, that's a viable solution if metadata make the index very big. The experience I had for very big indexes, i.e 500G indexes that won't fit a single host, one can use a secondary document storage database for document retrieval to speed things up. Alternatively (especially since solr 4.10), we could further tune the SOLR_HOME, sharding, caching etc to store the metadata in the same Solr instance. Chances are less dependencies if that scales well.