Closed esune closed 4 years ago
Something else to keep in mind and investigate is whether we have sufficient/proper indexes on the models/database to back the queries that are being executed - this will obviously not solve the data size issue, but will make increase performance when executing queries.
When requesting large credentialsets such as this one the processing time becomes very large, causing the request to time out.
making the timeout time longer doesn't seem to solve the issue as the APi pod runs out of memory and is killed while executing the query. This is likely because of how Django ORM executes queries and tries to fetch and cache the data for re-use (see https://docs.djangoproject.com/en/2.2/ref/models/querysets/#django.db.models.query.QuerySet.iterator).
To make this work better we could either:
credentialset
APi call to only return a limited number of records with each call.Either approach will require updates to the UI in order to support the updated APIs.
This issue ties in with #308 and #344
The issue is currently experienced with facility and well credentials, so not with "regular" OrgBook data. However, the possibility of large datasets in the credential registry seems like it could be a fairly common scenario.