The NIOSSLHandler can enter a spin loop in doUnbufferActions if writing data into BoringSSL fails with SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE.
One way these errors can happen is if a write into BoringSSL happens prior to the handshake completing. The NIOSSLHandler is explicit about starting the handshake: it's done in channel active and in handler added if the channel is already active.
However, the handshaking step is currently done without any state checking, so if the state is 'closed' (as it would be after channel inactive) then the handshake will still start. This can happen if channel inactive happens before channel active.
To reach the write loop there must be a buffered write and flush prior to the handshake step starting and the state isn't 'idle' or 'handshaking'. This can happen if a write and flush hapens while 'NIOSSLHandler' is in 'channelActive' (it forwards the 'channelActive' event before starting the handshake) and 'channelInactive' came first.
Modifications:
Early exit from 'doHandshakeStep' if the state isn't applicable for starting a handshake.
Don't buffer writes when the state is 'closed' as they'll never succeed and can be failed immediately.
If the write isn't succesful in 'doUnbufferActions' then either write to the network or try reading.
Only allow a limited number of spins through 'doUnbufferActions'
weissi
commented
2 months ago
Motivation:
The NIOSSLHandler can enter a spin loop in
doUnbufferActionsif writing data into BoringSSL fails with SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE.One way these errors can happen is if a write into BoringSSL happens prior to the handshake completing. The NIOSSLHandler is explicit about starting the handshake: it's done in channel active and in handler added if the channel is already active.
However, the handshaking step is currently done without any state checking, so if the state is 'closed' (as it would be after channel inactive) then the handshake will still start. This can happen if channel inactive happens before channel active.
To reach the write loop there must be a buffered write and flush prior to the handshake step starting and the state isn't 'idle' or 'handshaking'. This can happen if a write and flush hapens while 'NIOSSLHandler' is in 'channelActive' (it forwards the 'channelActive' event before starting the handshake) and 'channelInactive' came first.
Modifications:
Result: