CenterForOpenScience / SHARE

SHARE is building a free, open, data set about research and scholarly activities across their life cycle.
http://share-research.readthedocs.io/en/latest/index.html
Apache License 2.0
100 stars 58 forks source link

[SHARE-1025] Add "urgent" queues #760

Closed aaxelb closed 6 years ago

aaxelb commented 6 years ago

When there's a large backlog of ingest or indexing tasks, freshly pushed data (e.g. new OSF preprints) go to the back of the queue and may have to wait a long while before they're available in the search index.

This PR adds new "urgent" queues for each bottleneck which allow pushed data to jump to the head of the line. Also, tidies up the elasticsearch settings, including removing the various (completely unused) "priority" settings.

New queue names:

Deployment notes:

https://openscience.atlassian.net/browse/SHARE-1025

coveralls commented 6 years ago

Coverage Status

Coverage increased (+0.02%) to 79.205% when pulling d07058d2ff21690feea53dd9f0d434f471ad3bb6 on aaxelb:share-1025--urgent-queues into 84f31b833c10a32015492a4f43480b97667c4770 on CenterForOpenScience:develop.