Our golr queries make use of DISTINCT and UNION for joining and uniquifying paths and sub, pred, obj triples. This is memory intensive, and is possibly no longer required given the @jnguyenx's tinker graph refactor on the golr loader side. In testing this has improved memory usage, where memory usage is roughly equal to the graph size on disk, where as before we were using 2-3x size on disk.
Will first test this on the golr loader side and then with a full graph to see how this affects indexing speed.
Our golr queries make use of DISTINCT and UNION for joining and uniquifying paths and sub, pred, obj triples. This is memory intensive, and is possibly no longer required given the @jnguyenx's tinker graph refactor on the golr loader side. In testing this has improved memory usage, where memory usage is roughly equal to the graph size on disk, where as before we were using 2-3x size on disk.
Will first test this on the golr loader side and then with a full graph to see how this affects indexing speed.