When there's a large backlog of ingest or indexing tasks, freshly pushed data (e.g. new OSF preprints) go to the back of the queue and may have to wait a long while before they're available in the search index.
This PR adds new "urgent" queues for each bottleneck which allow pushed data to jump to the head of the line. Also, tidies up the elasticsearch settings, including removing the various (completely unused) "priority" settings.
New queue names:
Ingest tasks: ingest, ingest.urgent
Indexing to default search index: es-share, es-share.urgent
Indexing to TritonSHARE search index: es-triton-share, es-triton-share.urgent
Deployment notes:
Stop workers and wait until es-index and es-index-firehose queues are empty.
Coverage increased (+0.02%) to 79.205% when pulling d07058d2ff21690feea53dd9f0d434f471ad3bb6 on aaxelb:share-1025--urgent-queues into 84f31b833c10a32015492a4f43480b97667c4770 on CenterForOpenScience:develop.
When there's a large backlog of ingest or indexing tasks, freshly pushed data (e.g. new OSF preprints) go to the back of the queue and may have to wait a long while before they're available in the search index.
This PR adds new "urgent" queues for each bottleneck which allow pushed data to jump to the head of the line. Also, tidies up the elasticsearch settings, including removing the various (completely unused) "priority" settings.
New queue names:
ingest
,ingest.urgent
es-share
,es-share.urgent
es-triton-share
,es-triton-share.urgent
Deployment notes:
es-index
andes-index-firehose
queues are empty.https://openscience.atlassian.net/browse/SHARE-1025