search

mozilla / addons

☂ Umbrella repository for Mozilla Addons ✨
Other
127 stars 41 forks source link

More graceful git-extraction stuck recovery #1879

Open wagnerand opened 2 years ago

wagnerand commented 2 years ago

Sometimes, git-extraction just gets stuck/does not finish for a version, so the entry in the git-extraction queue stays with in_progress=True forever (until a human manually removes that entry from the queue). Meanwhile, submitted updates to that add-on trigger more additions to the git-extraction queue which are all skipped during the extraction task because there is one that is in progress already.

When an add-on extraction has been in progress for a long time (1h?), we should probably clear the in_progress flag to let the task try again. We might want to have a retry limit though, at which point the flag should no longer be cleared and a human should be alerted.

While an add-on has a git-extraction entry in the queue, we might also want limit further additions (caused by updates) to the queue for that add-on in order to avoid cluttering the queue. Although, if a human manages to follow up timely on above alert, there shouldn't be too many updates in the meantime.

┆Issue is synchronized with this Jira Task

KevinMind commented 6 months ago

Old Jira Ticket: https://mozilla-hub.atlassian.net/browse/ADDSRV-95

wagnerand commented 2 months ago

@diox @willdurand will we still need git-extraction once code-manager has been decommissioned? Related tickets:

diox commented 2 months ago

No. Once code-manager is decommissioned we will get rid of git extractions entirely (that's one of the main reasons why I want to move away from code-manager)