Process for identifying datastores that need column indexes

GetDKAN / dkan

DKAN Open Data Portal

https://dkan.readthedocs.io/en/latest/index.html

GNU General Public License v2.0

372 stars 172 forks source link

Process for identifying datastores that need column indexes #3651

Open grugnog opened 3 years ago

grugnog commented 3 years ago

User Story

Datastore queries (at least common ones that have previously been slow) are fast and efficient on server resources

Acceptance Criteria

Datastore indexes are added (and re-added if dataset re-uploads occur)
Datastore indexes account for real life usage (since it is impractical to anticipate every possible API query)

grugnog commented 3 years ago

Working is ongoing at https://git.drupalcode.org/project/indexer - I have code to parse slow query log data and identify compound indexes with the most value (most rows read in slow queries and highest cardinality) as well as identify existing indexes (so we don't duplicate those). Next I need to parse the tree and actually add or remove indexes as needed.

grugnog commented 3 years ago

So the main task of adding/removing indexes based on this analysis is complete. The remaining task is to (optionally) store these in config and automatically add them to new tables that match provided column name combinations.