Add a retry mechanic to requests that fail with MongoDB WriteConflict.

[ro-tex]

PULL REQUEST

Overview

There is a general problem with multi-step transactions when several of them overlap.

The problem goes like this:

1. txn1 reads value x, which currently sits at 5
2. txn1 makes some decisions based on that value
3. txn1 prepares to store its output to DB
4. txn2 changes the value of x from 5 to 6
5. txn2 commits
6. the engine realizes that the decisions made by txn1 are based on a now-changed value, i.e. they are invalid
7. the engine cancels txn1 and informs the client that they need to retry it, based on the new value of x

This PR adds the retry mechanic, so we can recover from such errors instead of directly returning them to the user.

Example for Visual Changes

Checklist

• [ ] All git commits are signed. (REQUIRED)
• [ ] All new methods or updated methods have clear docstrings.
• [ ] Testing added or updated for new methods.
• [ ] Verified if any changes impact the WebPortal Health Checks.
• [ ] Appropriate documentation updated.
• [ ] Changelog file created.

Issues Closed

Closes SKY-189

[linear[bot]]

SKY-189 MongoDB `WriteConflict`s

Some API calls fail with ``` { "message": "[failed to update; (WriteConflict) WriteConflict error: this operation conflicted with another operation. Please retry your operation or multi-document transaction.]" } ``` One example is the Stripe webhook but that's not the only affected endpoint. Maybe check the locking strategy used in TUS and use it when updating users?