The way query cancellation is implemented now, there is a race between cancelling the query in postgres, and potentially re-using that same connection for a future query. What can happen is:
A query [query1] is executed
The context for [query1] is cancelled
The watchCancel goroutine is triggered, which begins the query cancellation in postgres
Before postgres can finish cancelling [query1], it returns its result, doesn't see that it's being cancelled, and returns its result and is returned to the connection pool
This same connection is reused for another query, [query2]
The query cancellation from [query1] in postgres goes through
[query2] gets erroneously cancelled
At a few hundred transactions/s, we'd only see a few of these per day. Since applying the above patch, we haven't seen this issue for the past 3 days.
PR that fixes this issue here: https://github.com/lib/pq/pull/1000
The way query cancellation is implemented now, there is a race between cancelling the query in postgres, and potentially re-using that same connection for a future query. What can happen is:
At a few hundred transactions/s, we'd only see a few of these per day. Since applying the above patch, we haven't seen this issue for the past 3 days.