MolSSI / QCFractal

A distributed compute and database platform for quantum chemistry.
https://molssi.github.io/QCFractal/
BSD 3-Clause "New" or "Revised" License
144 stars 47 forks source link

Improve task queue performance #811

Closed bennybp closed 6 months ago

bennybp commented 6 months ago

Description

Simplifies the task queue. Now, the queue will store a timestamp used to sort tasks. By default, it is the time the row is added, however services will overwrite this with the created_on time of the service itself. This allows for services to finish relatively in order, without waiting for tasks from all other services to finish.

This behavior was already present, however the query to determine the next tasks was pretty gnarly and expensive. This is now replaced by a very simple, indexed query on one table.

Supersedes #810

Changelog description

Improves task queue performance

Status