[META] Generic Query Insights Framework

[ansjcy]

Is your feature request related to a problem? Please describe. This meta issue tracks all the features and tasks within the scope of the Query Insights framework. Currently, there are multiple areas we want to invest in to improve query visibility in OpenSearch:

• Query Insights plugin with collect, process, and export capabilities.
• Top queries insights by resource usage / latency.
• Metrics improvements, grouped by query categories / similar queries.
• Query Insights Dashboards with profiling capabilities.
• Recommendations based on query insights to improve search performance.

Describe the solution you'd like

Related OS RFCs:

• https://github.com/opensearch-project/OpenSearch/issues/11429
• https://github.com/opensearch-project/OpenSearch/issues/11186
• https://github.com/opensearch-project/OpenSearch/issues/12292
• https://github.com/opensearch-project/OpenSearch/issues/11008
• https://github.com/opensearch-project/OpenSearch/issues/11596
• https://github.com/opensearch-project/OpenSearch/issues/12084 (client side insights (UBI))
• https://github.com/opensearch-project/OpenSearch/issues/13357

Related OSD RFCs:

• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/5571
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/571 (We can potentially integrate the Query Insights Dashboard with Profiler UX experience.)

Child Meta Issues:

• [ ] https://github.com/opensearch-project/OpenSearch/issues/13419

Major Pending Issues:

• [x] https://github.com/opensearch-project/OpenSearch/issues/11762
• [x] https://github.com/opensearch-project/OpenSearch/issues/11520
• [x] https://github.com/opensearch-project/OpenSearch/issues/11295
• [x] https://github.com/opensearch-project/OpenSearch/issues/11431
• [x] https://github.com/opensearch-project/OpenSearch/issues/12399
• [x] https://github.com/opensearch-project/OpenSearch/issues/11296
• [ ] https://github.com/opensearch-project/OpenSearch/issues/12740
• [ ] https://github.com/opensearch-project/OpenSearch/issues/13369
• [ ] https://github.com/opensearch-project/OpenSearch/issues/13881
• [ ] https://github.com/opensearch-project/OpenSearch/issues/13357
• [ ] https://github.com/opensearch-project/OpenSearch/issues/14197
• [ ] https://github.com/opensearch-project/security/issues/4402
• [x] https://github.com/opensearch-project/documentation-website/issues/6180
• [x] https://github.com/opensearch-project/documentation-website/issues/6945
• [x] https://github.com/opensearch-project/documentation-website/issues/7325

Major Ongoing Tasks:

• [x] https://github.com/opensearch-project/OpenSearch/pull/11526
• [x] https://github.com/opensearch-project/OpenSearch/pull/11903
• [x] https://github.com/opensearch-project/OpenSearch/pull/11904
• [x] https://github.com/opensearch-project/OpenSearch/pull/12982
• [x] https://github.com/opensearch-project/OpenSearch/pull/13172
• [x] https://github.com/opensearch-project/OpenSearch/pull/13739
• [x] https://github.com/opensearch-project/OpenSearch/pull/13374
• [x] https://github.com/opensearch-project/security/pull/4006
• [x] https://github.com/opensearch-project/security/pull/4229
• [ ] https://github.com/opensearch-project/security/pull/4403
• [x] https://github.com/opensearch-project/documentation-website/pull/6261
• [x] https://github.com/opensearch-project/documentation-website/pull/7326

POC / Experimental PRs

• [ ] https://github.com/opensearch-project/OpenSearch/pull/11506
• [ ] https://github.com/opensearch-project/OpenSearch/pull/12449
• [ ] https://github.com/opensearch-project/OpenSearch/pull/12473
• [ ] https://github.com/opensearch-project/OpenSearch/pull/12529

Additional context Please add any tickets in the comments in case I'm missing anything.

[peternied]

[Triage - attendees 1 2 3 4] @ansjcy Thanks for filing this issue