search

bcgov / aries-vcr

Hyperledger Aries Verifiable Credential Registry (VCR) is a set of application level software components designed to accelerate the adoption of trustworthy entity to entity communications.
Apache License 2.0
78 stars 67 forks source link

Disable Solr caches #758

Closed WadeBarnes closed 8 months ago

WadeBarnes commented 9 months ago

As discussed some of the articles listed here caching is only useful when there are repeated queries for the same information, otherwise they can be detrimental to memory use. Queries in OrgBook tend to be random (Humans) or sequential (Spiders and Bots), so there are few, if any, cache hits. This could be, at least part of, the cause for the high memory usage we're seeing in prod.

Try disabling the caches as recommended to see the impact on memory use.

This issue is related to the investigation happening here; https://github.com/bcgov/orgbook-configurations/pull/137

WadeBarnes commented 9 months ago

I created a search-engine image off https://github.com/bcgov/aries-vcr/pull/759 and smoke tested in dev and test before deploying to prod. Performing some load testing before leaving it over the weekend to collect some metrics.

WadeBarnes commented 9 months ago

Disabling the Solr caches had no effect. If anything the OOM restart issues became worse following the change, though this is just speculation and the increased number of issue could be attributed to query patterns and volume. Some metrics collected here; https://github.com/bcgov/orgbook-configurations/pull/137#issuecomment-1764701163.

WadeBarnes commented 8 months ago

This actually helped reduce resource usage. It was actually unbound queries (https://github.com/bcgov/aries-vcr/issues/762) that were the main issue.