search

pelias / dashboard

Pelias dashboard built with the Dashing framework
7 stars 12 forks source link

feat(layers): Use aggregation to count records #14

Closed orangejulius closed 5 years ago

orangejulius commented 5 years ago

This project previously depended on Elasticsearch types in order to provide a count of the number of records in each layer.

Elasticsearch types will be going away in Elasticsearch 6, and are already deprecated in ES5.

Instead of relying on types, the count of records per layer is now created using a terms aggregation similar to the source and layers autodetection code recently added to pelias/api.

This is one of the last few roadblocks for dropping Pelias's use of types completely and merging PRs like https://github.com/pelias/schema/pull/293 that will drastically simplify our schema.

Connects https://github.com/pelias/pelias/issues/461 Connects https://github.com/pelias/pelias/issues/719

orangejulius commented 5 years ago

Note that in my testing, this change has serious performance implications for anyone still using Elasticsearch 2. The aggregations queries generated by the dashboard will not be cached, are fairly expensive to run, and are run frequently.

If you have a large Elasticsearch index (10M records or more), you will notice a serious performance impact from this change unless you upgrade to Elasticsearch 5, which you should do!