Write reference/runbook for codecov-api "sync vs. async" root cause of bugs

[suejung-sentry]

This task captures investigation into where development tripping up on sync vs. async has been the root cause of bugs in codecov-api.

Definition of Done - have some sort of technical reference of why/when this error occurs so developers in codecov-api can watch out for falling in the "trap" in their own code and when reviewing others' PRs. Also would be nice to have an on-call runbook for tips on debugging or recognizing when this is the root cause of the issue & how to fix (hot-fix or long-term fix).

If there are any outcomes from the investigation of a more systemic solution (linter? type checker?), documenting those as well so they can get picked up and done in some future work.

Past examples:

• https://github.com/codecov/codecov-api/pull/783
  • This is the instance where Nora first explained to me about encountering the issue. She mentioned the error surfaced as You cannot call this from an async context - use a thread or sync_to_async.

Some other old instances I saw in slack:

• https://github.com/codecov/worker/pull/324/files
• https://github.com/codecov/engineering-team/issues/2223

Here's a more recent one that Michelle fixed for us: https://github.com/codecov/codecov-api/pull/794 (guessing it cost the team at least a dozen developer-hours to figure out / fix)

[suejung-sentry]

Update - some learnings to date (to be collated after investigation done)

• We use Django ORM in api to access postgres. These queries are always synchronous operations.
• We use GraphQL ariadne python library to manage our GraphQL API layer (TODO - which parts?). This runs as asynchronous operations.
• When ariadne makes an api request that involves db interaction, it expects an async function (i.e., a coroutine) that can be awaited without blocking the event loop. This can be accomplished if the db call is wrapped in Django's sync_to_async utility (from asiref.sync lib)
• To catch instances of this being missed before it reaches prod, we can potentially adopt this linter flake8-async which has lint rules we can implement in ruff. Ruff is already set up in codecov-api (thanks Ajay!) so it would just be a matter of adding in the lint rule, seeing if it solves our problem on this, fix any existing rule violations, & make sure it's integrated to our CI to prevent this in future development

More details TBD as we learn more about this