Stack overflow at random times(might be related to DDOS attacks)

[aviv0099]

Can happen instantly when I start my server, it worked great for a year with no problem but now in the last 24 hours it crashed 10+ times with this weird error: `Stack overflow. at Hazel.MessageReader.GetSized(Int32) at Hazel.Udp.UdpConnectionListener.StartListeningForData() at Hazel.Udp.UdpConnectionListener.ReadCallback(System.IAsyncResult) at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback) at System.Net.Sockets.Socket.BeginReceiveFrom(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.EndPoint ByRef, System.AsyncCallback, System.Object) at Hazel.Udp.UdpConnectionListener.StartListeningForData() at Hazel.Udp.UdpConnectionListener.ReadCallback(System.IAsyncResult) at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback) at System.Net.Sockets.Socket.BeginReceiveFrom(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.EndPoint ByRef, System.AsyncCallback, System.Object)

It goes like this for around 10K lines! then at the end it says

at Hazel.Udp.UdpConnectionListener.StartListeningForData() at Hazel.Udp.UdpConnectionListener.Start() at Prison_Life_Server.ServerProgram.Init(Boolean, Boolean)`

Any idea how can I fix it or can start to investigate? I suspect it might be related to DDOS attack but it happens instantly so I can't even track what message I'm getting. on Linode I saw at one point I get around 1GBPS for few moments.

And I also have a question what is the benefit of using "ThreadLimitedUdpConnectionListener" instead of the regular one?

[willardf]

You should definitely look into the ThreadLimitedUdpConnectionListener. It would easily avoid this stack overflow.

The big difference between the two listeners is:

• ThreadLimited dedicates 2 threads in a tight loop to read and write the socket, then allows you to decide how many worker threads handle the queued messages afterwards.
• UdpConnectionListener uses Socket.BeginReceive/Send which uses pooled threads to read/write from the socket. There's no way to control how many threads are in use, which makes it more susceptible to DoS attacks. Additionally, it uses recursion to restart the listening process. This shouldn't ever cause a stack overflow. I don't think I've ever seen one, but it sure seems like I'm wrong.

To be honest, I think this might be the push I need to remove UdpConnectionListener from the project. I'm pretty confident that ThreadLimitedConnectionListener is better in every way, but I didn't want to remove the old version because I wasn't sure how many people are using it. But this is such a basic bug that could be avoided, so perhaps it's time to make the switch.

[aviv0099]

Thank you for the response, you are awesome! I changed a few hours ago to the ThreadLimitedUdpConnectionListener and everything seems to be fine for now. How many workers do you recommend to put it on? currently I set it to 2

Another question, why did you choose to use DTLS among us when it's more expensive to run?

[willardf]

Awesome, glad to hear it! If a single core VM or something, then 2 is pretty good. If the hardware has more cores, then the number of cores is a good start. The best value depends on how you're handling the messages, so you just have to test and profile.

DTLS is encrypted UDP, which is a requirement for certain platforms, so basically we had to.

[aviv0099]

Make sense, thank you, wish you the greatest success!