search

IHTSDO / snowstorm

Scalable SNOMED CT Terminology Server using Elasticsearch
Other
208 stars 83 forks source link

Large number of ES scroll contexts created during import #44

Closed kaicode closed 4 years ago

kaicode commented 5 years ago

During the initial snapshot import of the SNOMED CT RF2 file there are up to 10K Elasticsearch scroll contexts open at once. This may be slowing down the import. We should investigate if these search contexts can be closed more quickly in code, perhaps in the spring-data-elasticsearch layer, rather than relying on the scroll context timeout which is likely at the moment.

See the open_contexts stat when importing here http://localhost:9200/_nodes/stats

Thanks to @rorydavidson for finding this.

kaicode commented 4 years ago

This is resolved in Snowstorm release 4.7.1.