Closed rsimon closed 6 years ago
This looks more like an issue of indexing records with large/complex geometries, rather than the conflation issue described above. Added the fix recommended at https://github.com/elastic/elasticsearch/issues/22087#issuecomment-334759601. This seems to improve things considerably.
While out-of-memory errors due to large geometries seem to be resolved, heavy conflation seems to be an issue, too. Just encountered this now with the ToposText gazetteer dataset.
It looks as if this is indeed nothing on the application side. At some point, ElasticSearch simply seems to get clogged. As a workaround, I introduced a 5s wait as soon as indexing a single item takes longer than 1s.
Other than that, the only thing we can likeyl do is upgrade to newer versions of ES. (Which was so far not possible due to Play framework version conflicts, but will be one of the first major milestones in the next dev face fingers crossed)
Data that will cause large amounts of conflation (in other words, gazetteers) currently crash the index, if it is large already (>500k items). I have yet to track down what the actual problem is (memory?). We will need to find a way around this.
In principle, Peripleo code runs the ingest sequentially (per dumpfile), and waits for individual index operations to finish. Also, it's clearly the index that crashes, not the application. Therefore, future upgrades to newer ElasticSearch versions might resolve the issue. Either way, we'll need to find out.