SciGraph / golr-loader

Convert SciGraph queries into json that can be loaded by Golr
Apache License 2.0
1 stars 3 forks source link

Optimizations to golr loader, removes MapDB #48

Closed kshefchek closed 5 years ago

kshefchek commented 5 years ago

The golr loader currently takes 30 hours to load all documents excluding optimizing, moving files, setup/teardown on the 201902 release of scigraph, using the queries on https://github.com/monarch-initiative/monarch-cypher-queries/tree/60a3cf

This update improves the load time to 16 and a half hours with the following updates:

The downside is that we monopolize all the memory on one of our servers for ~8 hours.

See the accompanying PRs: https://github.com/monarch-initiative/monarch-cypher-queries/pull/33 https://github.com/monarch-initiative/solr-docker-monarch-golr/pull/1