Is your feature request related to a problem? Please describe
As we add more query insights services like top n queries by different metrics types and top queries shapes, we are introducing higher memory usage to store all the metrics. We should provide the ability for the user to set memory budget for query insights features to avoid over-using memory on each node.
Describe the solution you'd like
We can provide separate, but generic configuration endpoints to set limits on memory usages for diffrent query inisght services. For example, for top n queries, we can provide something like:
to set the maximum memory that the top n queries by latency service can use to 5mb.
Related component
Search:Query Insights
Describe alternatives you've considered
We can also consider providing only one configuration endpoint to set limit on resource usage for all the query insights components. It would make the configuration easier but will also take away the customization for different query insights features. If one feature is using too much memory it would starve other query insights services.
[Triage - attendees 12345678]
@ansjcy Thanks for creating this issue. Please work with maintainers in the area to see that this approach will not introduce too much overhead for cluster administration.
peternied
commented
5 months ago
Is your feature request related to a problem? Please describe
As we add more query insights services like top n queries by different metrics types and top queries shapes, we are introducing higher memory usage to store all the metrics. We should provide the ability for the user to set memory budget for query insights features to avoid over-using memory on each node.
Describe the solution you'd like
We can provide separate, but generic configuration endpoints to set limits on memory usages for diffrent query inisght services. For example, for top n queries, we can provide something like:
to set the maximum memory that the top n queries by latency service can use to 5mb.
Related component
Search:Query Insights
Describe alternatives you've considered
We can also consider providing only one configuration endpoint to set limit on resource usage for all the query insights components. It would make the configuration easier but will also take away the customization for different query insights features. If one feature is using too much memory it would starve other query insights services.
Additional context
Please refer to the meta issue: https://github.com/opensearch-project/OpenSearch/issues/11522