Open hardbyte opened 5 years ago
Given the pr for supporting run deletion introduced a cascading delete on the permutation mask table, which is on the flask error, it could be the database for the integration tests of the develop branch wasn't cleanly initiated eg due to a persistent volume.
This race condition occured while running the integration tests here.
At first glance it seems like the celery worker correctly threw a DB exception in the
compute_filter_similarity
task, leading to the callback throwing an exception... so far this is fine. I note that the celery worker then tries to write to the runs table again to mark the run as failed:Over in the flask app we queue a task to delete any similarity files then start a db transaction to delete the run. It is conceivable that the delete files task finishes before a file is written.
Lastly in the flask app, we don't catch the DB integrity error and end up giving the user a 500 response :-/
Perhaps this could be addressed more like project deletion - by setting the run as
deleted
in the database instead of deleting the row.Note I didn't get to see the high memory worker logs - see #418
Logs
Integration test failure:
Flask:
Worker: