The OmniCorp postgres DB stops servicing pubmed id count requests for large workloads.
ranker creates a job/curie to gather the pubmed id counts. in situations where there are a large number of curies to lookup (>1-2k?) the DB may go unresponsive. presumably this is occurring because the DB is overwhelmed with the nearly instantaneous flood of SELECT COUNT(pubmed_id) requests.
a number of things were attempted to alleviate/isolate the issue. most of these enhanced performance but did not correct the Issue:
increased the number of CPUs allocated to the DB instance (currently set at 5, some railing of all CPUS still occurs)
increased the available memory to the DB instance.
added table indexes. there are no longer table scans occurring (only index scans). this had the greatest positive impact on performance.
DB settings should be inspected to see if there is a way to optimize for this scenario. in the interim it may be advantageous to somehow throttle the flood of select statements.
The OmniCorp postgres DB stops servicing pubmed id count requests for large workloads.
ranker creates a job/curie to gather the pubmed id counts. in situations where there are a large number of curies to lookup (>1-2k?) the DB may go unresponsive. presumably this is occurring because the DB is overwhelmed with the nearly instantaneous flood of SELECT COUNT(pubmed_id) requests.
a number of things were attempted to alleviate/isolate the issue. most of these enhanced performance but did not correct the Issue:
DB settings should be inspected to see if there is a way to optimize for this scenario. in the interim it may be advantageous to somehow throttle the flood of select statements.