In many places we update the state of jobs or executions, then create an evaluation, and then enqueue the evaluation to the broker. All of these operations are not atomic and not in a single transaction. This can result in a lot of edge cases and potentially dangling or stale jobs and executions due to partial update of their state, but failure to trigger their re-evaluation.
We need to look into ways to do these operations atomically in a single transaction. This is might be easier to update the job state and create an evaluation in a single transaction as they are part of the same datastore, but the evaluation broker has its own in-memory state, and we need to figure out a way to implement state watchers where we don't need to explicitly enqueue evaluations, but allow the broker to consume state events and enqueue an evaluation whenever one is created
In many places we update the state of jobs or executions, then create an evaluation, and then enqueue the evaluation to the broker. All of these operations are not atomic and not in a single transaction. This can result in a lot of edge cases and potentially dangling or stale jobs and executions due to partial update of their state, but failure to trigger their re-evaluation.
We need to look into ways to do these operations atomically in a single transaction. This is might be easier to update the job state and create an evaluation in a single transaction as they are part of the same datastore, but the evaluation broker has its own in-memory state, and we need to figure out a way to implement state watchers where we don't need to explicitly enqueue evaluations, but allow the broker to consume state events and enqueue an evaluation whenever one is created