bcgov / aries-vcr

Hyperledger Aries Verifiable Credential Registry (VCR) is a set of application level software components designed to accelerate the adoption of trustworthy entity to entity communications.
Apache License 2.0
78 stars 70 forks source link

Timeout when requesting large credentialset #397

Closed esune closed 4 years ago

esune commented 4 years ago

When requesting large credentialsets such as this one the processing time becomes very large, causing the request to time out.

making the timeout time longer doesn't seem to solve the issue as the APi pod runs out of memory and is killed while executing the query. This is likely because of how Django ORM executes queries and tries to fetch and cache the data for re-use (see https://docs.djangoproject.com/en/2.2/ref/models/querysets/#django.db.models.query.QuerySet.iterator).

To make this work better we could either:

Either approach will require updates to the UI in order to support the updated APIs.

This issue ties in with #308 and #344

The issue is currently experienced with facility and well credentials, so not with "regular" OrgBook data. However, the possibility of large datasets in the credential registry seems like it could be a fairly common scenario.

esune commented 4 years ago

Something else to keep in mind and investigate is whether we have sufficient/proper indexes on the models/database to back the queries that are being executed - this will obviously not solve the data size issue, but will make increase performance when executing queries.