Issue
We have been noticing for a while servers that focus on the webrtc connections randomly becoming unresponsive. This unresponsive state happens anywhere from a couple minutes to ten minutes. If you are looking at the charts the CPU drops off and then once it recovers you see a spike as all the registrations are flooding back in. We have also seen cases where the server never recovers until we restart it. In every case the server seems to keep doing stuff but new registrations are blocked.
I was finally able to recreate something that looks very similar to what we are seeing.
The test environment
Drachtio Server 0.8.24
Around 60 SIP connections but able to do it with around 10
How to recreate
Open up 8 terminal windows. In each terminal run the command openssl s_client -connect URL_OF_WEBRTC_SERVER:WEBRTC_PORT right after one another. Once the command stops in one window, run it again. This timing changes and seems to be a perfect timing thing. I had it happen after 9 commands and I had it after 20ish commands. Once it happens all the terminals will be blocked in a "CONNECTED" state besides one which will say read R BLOCK.
At this point you will see on the Drachtio server addAppTransaction and addApiRequest size increase. You will also see registrations will start failing to happen.
To clear it from this state go back to the terminal that says read R BLOCK and end that connection. All the other terminals will start processing again and the registrations will start working again. I was able to recreate this every time I ran the test.
Logs
Around where the block starts 2024-01-24 22:05:29.120311
Around where the block ends 2024-01-24 22:07:21.284546
Issue We have been noticing for a while servers that focus on the webrtc connections randomly becoming unresponsive. This unresponsive state happens anywhere from a couple minutes to ten minutes. If you are looking at the charts the CPU drops off and then once it recovers you see a spike as all the registrations are flooding back in. We have also seen cases where the server never recovers until we restart it. In every case the server seems to keep doing stuff but new registrations are blocked.
I was finally able to recreate something that looks very similar to what we are seeing.
The test environment
How to recreate Open up 8 terminal windows. In each terminal run the command
openssl s_client -connect URL_OF_WEBRTC_SERVER:WEBRTC_PORTright after one another. Once the command stops in one window, run it again. This timing changes and seems to be a perfect timing thing. I had it happen after 9 commands and I had it after 20ish commands. Once it happens all the terminals will be blocked in a "CONNECTED" state besides one which will sayread R BLOCK.At this point you will see on the Drachtio server
addAppTransactionandaddApiRequestsize increase. You will also see registrations will start failing to happen.To clear it from this state go back to the terminal that says
read R BLOCKand end that connection. All the other terminals will start processing again and the registrations will start working again. I was able to recreate this every time I ran the test.Logs
sip-webrtc-redacted.txt