Add Node Self Protection from OOM's

opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data

https://opensearch.org/docs/latest/search-plugins/knn/index/

Apache License 2.0

152 stars 113 forks source link

Add Node Self Protection from OOM's #1762

Open kkmr opened 3 months ago

kkmr commented 3 months ago

Today, it is up to the customer to provision appropriate number of nodes so that the cluster/domain has sufficient memory to handle their load. If customer under provisions the nodes, there is a potential for the nodes to OOM. We should consider adding the following functionality:

A mechanism to measure the total memory in use in the system (like RSS)
A mechanism to throttle graph loads when memory usage exceeds a certain threshold.

vamshin commented 3 months ago

@kkmr we do provide memory usage details part of _knn/stats https://opensearch.org/docs/latest/search-plugins/knn/api/#usage. This usage is based on graph sizes and not on RSS

We also have Circuit breakers in place to limit the number of graphs that can be in memory. If Circuit breaker(CB) is hit, we evict cache based on LRU. You should be able to see status of CB from stats https://opensearch.org/docs/latest/search-plugins/knn/api/#stats