imer Linux dropping connections after 100+ CCU because buffer sizes

[miwarnec]

from discord:

to further this: https://stackoverflow.com/a/2290848 looking at the /proc/net/udp6 one I can see many drops while doing the stress test with 100 clients: https://share.dl.je/2021/06/2021-06-13_19-00-55_UwDiLFug52.mp4 increasing the kernel bufffers (/proc/sys/net/core/wmem_default/max and /proc/sys/net/core/rmem_default/max) by 100x (as a quick validation) fixed the issue - no more drops! this is also reflected by the c# socket .Receive-/.SendBufferSize KCP Receive/Send Buffer is: 21299200/21299200 - the _max doesn't change those. need to adjust _default for that

500 ccu on linux kcp: https://share.dl.je/2021/06/2021-06-13_19-26-56_5ZDNb9zwiv.png

[miwarnec]

also imer:

500 ccu on linux kcp: https://share.dl.je/2021/06/2021-06-13_19-26-56_5ZDNb9zwiv.png [19:31] imer: checking if we can set the socket.Receive/SendBufferSize above the _max kernel param [19:33] imer: KCP Receive/Send Buffer is 212992/212992 by default.. increasing KCP Receive/Send Buffer is now 425984/425984 mh, It didnt set it to what I asked it to

        Debug.LogError($"KCP Receive/Send Buffer is {socket.ReceiveBufferSize}/{socket.SendBufferSize} by default.. increasing");
        socket.ReceiveBufferSize = 2129920;
        socket.SendBufferSize = 2129920;
        Debug.LogError($"KCP Receive/Send Buffer is now {socket.ReceiveBufferSize}/{socket.SendBufferSize}");

[19:33] imer: yep, thats not enough - dropping packets again [19:35] imer: alright, lets see if we can do 1000 connections with large buffers for the fun of it [19:47] imer: yes, needs > 20mbyte buffers though - just set it to 200mb and that works :grin: it's the receive buffer thats the issue - guess it's extremely bursty due to the local network or something? super laggy, it's not exploding though https://share.dl.je/2021/06/2021-06-13_19-48-04_eB1extWmS4.mp4 900ms/tick https://share.dl.je/2021/06/2021-06-13_19-48-51_qvhSPBQCLc.png

[miwarnec]

options:

• increase buffers in C# as much as OS allows
• add a warning if buffers get too full that connections might drop?
• re-evaluate after bitpacking. might reduce buffer sizes enough.
• poll more often? the more often we pull, the lower the receive buffer size remains
  • => perhaps in a thread
• use more than one socket for poll: supposedly doesn't work (imer tried)?
• include 'increase socket buffers' in our hosting guides?

[angelskieglazki]

@miwarnec Hello.

Is it possible to test the server on Linux and client that you used? where can I get it