aiarena / aiarena-web

A website for running the aiarena.net ladder.
https://aiarena.net
GNU General Public License v3.0
93 stars 16 forks source link

Stats generation job fails entirely on deadlock detection #583

Open lladdy opened 1 year ago

lladdy commented 1 year ago

Occasionally, a deadlock will occur when attempting to lock a competition participation. Instead of the entire stats gen job falling over, it should at minimum skip the competition participation in question.

More ideally it should wait and then retry whatever caused the deadlock, with some max retry attempt.

lladdy commented 1 year ago

This has been improved with an update but still results in bots not having their stats updated for up to ~a week.

lladdy commented 1 year ago

Relevant stats job code is here, for anyone interested in trying to refine it: https://github.com/aiarena/aiarena-web/blob/staging/aiarena/core/management/commands/generatestats.py