Closed GNMoseke closed 1 year ago
@gwynne @fabianfett Does the fix (workaround?) described seem to be a useful change to make?
I'll let @fabianfett take this one, that's all his code 😅
@GNMoseke Looking into this. There is multiple things going on here:
@Austinpayne, @GNMoseke #379 should address the crash that you are seeing. Would you mind validating this fix?
Please reopen this issue, if above fix doesn't solve your problem.
Describe the bug
Bug has been previously described here
We are seeing a crasher from
PostgresNIO.ConnectionStateMachine.closeAndCleanup(_:)
where postgres-nio attempts to re-use an already closed connection. The failure happens about 80% of the time in a given unit test, and seems to stem from a quick sequence of Failed Insert -> Select -> Delete requests.To Reproduce
The reproducer linked in the swift forum post above is a good place to start. I've also gathered logs from the (rare) success case and much more frequent case below:
Success Logs
^ As you can see in this example, the failed insert query properly dispatches an error message to the state machine, which closes the connection, releases it, sets the channel to inactive, and then properly prunes it. The next statement, the
SELECT
, is then properly dispatched onto a newpsql_connection_id
of 4.Failure Logs
^ Here, instead, we see that the
psql_connection_id
3 that received an error is released and set to inactive, but then is immediately given a.readyForQuery(.idle)
on the same connection ID, which results in the crash.I have a suspicion and a workaround:
state
of theConnectionStateMachine
is not placed in a read-write lock, and there is a race condition where the state is set toreadyForQuery
, receives a message, and then some other thread sets it toclosed
when an error is being handled, resulting in the assertion failure.ConnectionStateMachine.swift
fromto
Which is the same as in the closing case. This seemed to get rid of the problem and allow the channels to catch up and resolve themselves, and further queries executed without issue. However, we were still able to enter an "unreachable" state, which seems to be a bug.
Expected behavior
Connection error handling should fully clean up and close its connection without additional messages being allowed on that connection.
Environment
Seen in both Xcode 14.0.1 using swift 5.7+ and on ubuntu 20.04 using swift 5.7+ postgres-nio
1.15.0
Additional context
Again, my suspicion is the lack of a lock around the ConnectionStateMachine's
State
, but I am unsure if that was done intentionally. I also wonder if it may just be that the bad state can happen occasionally and the solution should be to handle it gracefully rather than with an assertion failure.