suejung-sentry
commented
1 month ago Update - some learnings to date (to be collated after investigation done)
- We use Django ORM in
apito access postgres. These queries are always synchronous operations. - We use GraphQL ariadne python library to manage our GraphQL API layer (TODO - which parts?). This runs as asynchronous operations.
- When ariadne makes an api request that involves db interaction, it expects an async function (i.e., a coroutine) that can be awaited without blocking the event loop. This can be accomplished if the db call is wrapped in Django's
sync_to_asyncutility (fromasiref.synclib) - To catch instances of this being missed before it reaches prod, we can potentially adopt this linter
flake8-asyncwhich has lint rules we can implement inruff. Ruff is already set up in codecov-api (thanks Ajay!) so it would just be a matter of adding in the lint rule, seeing if it solves our problem on this, fix any existing rule violations, & make sure it's integrated to our CI to prevent this in future development
This task captures investigation into where development tripping up on
syncvs.asynchas been the root cause of bugs incodecov-api.Definition of Done - have some sort of technical reference of why/when this error occurs so developers in
codecov-apican watch out for falling in the "trap" in their own code and when reviewing others' PRs. Also would be nice to have an on-call runbook for tips on debugging or recognizing when this is the root cause of the issue & how to fix (hot-fix or long-term fix).If there are any outcomes from the investigation of a more systemic solution (linter? type checker?), documenting those as well so they can get picked up and done in some future work.
Past examples:
You cannot call this from an async context - use a thread or sync_to_async.Some other old instances I saw in slack:
Here's a more recent one that Michelle fixed for us: https://github.com/codecov/codecov-api/pull/794 (guessing it cost the team at least a dozen developer-hours to figure out / fix)