openflighthpc / flight-control

0 stars 0 forks source link

Avoid stuck action logs if multiple changes in quick succession #67

Closed timalces closed 1 year ago

timalces commented 1 year ago

Aims to resolve #61

After some investigation, it was found that if an instance is switched on and off again (or vice verse) in quick succession, there is a small chance that the related action logs will become stuck in a 'pending' state, if instance status is not checked & recorded before the second action completes. E.g.

This PR resolves the problem by adding last_checked to InstanceLogs, set every time the task instance_logs:record is run. This is used instead of updated_at when determining if an action log's target status has been reached. The existing complete_previous method/workflow ensures any earlier pending action logs for the same instance are also marked as completed.

To prevent this also resulting in the new data check being triggered every time instance statuses are checked, a new field last_status_change is also added to InstanceLogs, and this used for the data check rather than updated_at

timalces commented 1 year ago

Merging this as appears to have fixed the problem.