We have observed on Mainnet that when the database is having errors (such as performance issues leading to timeouts acquiring a connection), some fetches never return, because we erroneously believe an object is missing and wait for, when really it was just temporarily inaccessible and no one is fetching it.
This PR:
Makes low-level fetching-related functions fallible, so we can more easily distinguish between an object being missing vs temporarily inaccessilbe
Retries fetch initialization until we successfully trigger the fetch
The guarantee that this change upholds is that every fetch will eventually resolve as long as the requested object is either fetchable or exists in the database already.
See commit messages, comments and new tests for additional details.
We have observed on Mainnet that when the database is having errors (such as performance issues leading to timeouts acquiring a connection), some fetches never return, because we erroneously believe an object is missing and wait for, when really it was just temporarily inaccessible and no one is fetching it.
This PR:
The guarantee that this change upholds is that every fetch will eventually resolve as long as the requested object is either fetchable or exists in the database already.
See commit messages, comments and new tests for additional details.