search

bcgov / aries-vcr

Hyperledger Aries Verifiable Credential Registry (VCR) is a set of application level software components designed to accelerate the adoption of trustworthy entity to entity communications.
Apache License 2.0
78 stars 67 forks source link

Investigate how Aries-VCR is performing paging when querying Solr #757

Open WadeBarnes opened 9 months ago

WadeBarnes commented 9 months ago

As discussed some of the articles listed here certain workarounds for paging can be problematic for Solr. Particular search queries seem to have an effect on the search-engine container (Solr), causing it to use a significant amount of memory.

Example queries (note the high page numbers):

/api/v4/search/topic?format=json&ordering=-revoked_date&page=149713
/api/v4/search/topic?format=api&ordering=revoked_date&page=2721
/api/v4/search/credential/facets?format=api&page=318768
/api/v4/search/credential?ordering=effective_date&page=318772

This issue is related to the investigation happening here; https://github.com/bcgov/orgbook-configurations/pull/137

WadeBarnes commented 9 months ago

Try this first; https://github.com/bcgov/aries-vcr/issues/758

WadeBarnes commented 9 months ago

~Disabling the Solr caches has been ineffective; https://github.com/bcgov/aries-vcr/issues/758#issuecomment-1764718361, https://github.com/bcgov/aries-vcr/issues/758~

I think it's time we get together to discuss next steps on how to proceed. A good start would be to assign someone to review, and then we can go over the tools we have for monitoring and what we've discovered about the issue so far.

WadeBarnes commented 8 months ago

The issues we were seeing with Solr were caused by unbound queries (https://github.com/bcgov/aries-vcr/issues/762) and likely made worse by the paging implementation.