mozilla / addons

☂ Umbrella repository for Mozilla Addons ✨
Other
125 stars 41 forks source link

Git-extraction fails for long-running tasks due to database connection timeout #1884

Open wagnerand opened 2 years ago

wagnerand commented 2 years ago

When git-extraction takes a long time due to the amount or size of packed files, the database connection is closed before the tasks finishes. This makes updating the database and inserting the git-hash fail.

https://sentry.io/organizations/mozilla/issues/3190120144/

┆Issue is synchronized with this Jira Task

diox commented 2 years ago

QA: check that git extractions are working by ensuring an add-on contents is available in code-manager after upload & after signing (note that it's done asynchronously though, and it can take a while depending on the add-on size - ideally test with a small and a bigger add-on with lots of files).

ioanarusiczki commented 2 years ago

@diox I did not find issues on AMO Stage, however I'll try again a couple of checks tomorrow too.

ioanarusiczki commented 2 years ago

@diox What I've uploaded on -dev and -stage, -> 5MB and 10 MB addons with many files have been extracted and displayed.

diox commented 2 years ago

We had a different failure that we need to investigate. An extraction exceeded the soft time limit, and then the error handler itself triggered the lost connection to MySQL error when trying to remove the in-progress entry from the queue.

It appears that the add-on was eventually removed though... Need to log into logs to see what happened exactly, and figure out next steps from there.

Regardless of the git extraction taking too long, we probably want a better solution for MySQL connection issues in long running tasks. I'm not sure what, maybe find a way to mark the connection as needing open + autocommit + close for each query somehow.

KevinMind commented 4 months ago

Old Jira Ticket: https://mozilla-hub.atlassian.net/browse/ADDSRV-100

wagnerand commented 2 weeks ago

@diox @willdurand will we still need git-extraction once code-manager has been decommissioned? Related tickets: