count_indexed_fields() failing for large ES indices

MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform

MIT License

26 stars 11 forks source link

count_indexed_fields() failing for large ES indices #254

Closed ghukill closed 5 years ago

ghukill commented 5 years ago

For a current ES index with 7.5million+ documents, the CombineJob method count_indexed_fields is failing, due to ES timeouts.

However, limiting the number of files added for aggregation helps dramatically, suggesting it would be possible to split up the aggregation requests if an index is particularly large. Or there are lots of fields.

ghukill commented 5 years ago

Fixed/fixing:

query ES field by field to avoid timeouts
saved concatenated results to job_details

Does not solve problem for Published Records, but those are about to get overhauled as well.