jsjiang
commented
3 months ago The link checker job by proc-link-chekcer.py might have contributed to this long waits query:
SELECT `ezidapp_searchidentifier` . `id` , `ezidapp_searchidentifier` . `identifier` , `ezidapp_searchidentifier` . `owner_id` , `ezidapp_searchidentifier` . `status` , `ezidapp_searchidentifier` . `target` , `ezidapp_searchidentifier` . `isTest` FROM `ezidapp_searchidentifier` WHERE `ezidapp_searchidentifier` . `identifier` > ? ORDER BY `ezidapp_searchidentifier` . `identifier` ASC LIMIT ?
SELECT `ezidapp_searchidentifier`.`id`, `ezidapp_searchidentifier`.`identifier`, `ezidapp_searchidentifier`.`owner_id`, `ezidapp_searchidentifier`.`status`, `ezidapp_searchidentifier`.`target`, `ezidapp_searchidentifier`.`isTest` FROM `ezidapp_searchidentifier` WHERE `ezidapp_searchidentifier`.`identifier` > 'ark:/88122/gnch0276' ORDER BY `ezidapp_searchidentifier`.`identifier` ASC LIMIT 1000
search_identifier_model = django.apps.apps.get_model('ezidapp', 'SearchIdentifier')
siGenerator = self.harvest(
search_identifier_model,
["identifier", "owner", "status", "target", "isTest"],
lambda si: si.isPublic
and not si.isTest
and si.target != si.defaultTarget
and si.owner_id not in self._permanentExcludes,
)
The
proc-link-checkerbackground job is memory intensive on the RDS server. The core SQL query is identified as one of the long running queries. Refactor the process to make the SQL queries more efficient. Or consider to use OpenSearch as data sources.Also, the job reschedules itself to run at the same time a week later.
Refactor items: