Server crash

[Marctraider]

Hey again! Not sure if this is linked to latest code, but I was doing some SQM/QOS testing and restricted one WAN interface on client side to like 100kbit/s.

After some load testing the server crashed. That's a first in a like over a week! Nothing fundamentally changed on the VPS side. No crash client side.

time="2019-11-29T22:07:23+01:00" level=info msg="New client connected: '62.131.70.190:49954'"
time="2019-11-29T22:07:25+01:00" level=info msg="New client connected: '62.131.70.190:43027'"
time="2019-11-29T22:07:27+01:00" level=info msg="New client connected: '62.131.70.190:38965'"
time="2019-11-29T22:07:36+01:00" level=info msg="New client connected: '62.131.70.190:60572'"
time="2019-11-29T22:07:42+01:00" level=info msg="New client connected: '62.131.70.190:39355'"
time="2019-11-29T22:07:43+01:00" level=info msg="New client connected: '62.131.70.190:36497'"
time="2019-11-29T22:07:44+01:00" level=info msg="Client '62.131.70.190:41589' timed out"
time="2019-11-29T22:07:44+01:00" level=info msg="New client connected: '62.131.70.190:33321'"
time="2019-11-29T22:07:45+01:00" level=info msg="New client connected: '62.131.70.190:53847'"
time="2019-11-29T22:07:46+01:00" level=info msg="Client '62.131.70.190:49954' timed out"
time="2019-11-29T22:07:46+01:00" level=info msg="New client connected: '62.131.70.190:47163'"
time="2019-11-29T22:07:47+01:00" level=info msg="Client '62.131.70.190:43027' timed out"
time="2019-11-29T22:07:47+01:00" level=info msg="New client connected: '62.131.70.190:52890'"
time="2019-11-29T22:07:48+01:00" level=info msg="New client connected: '62.131.70.190:38526'"
time="2019-11-29T22:07:49+01:00" level=info msg="New client connected: '62.131.70.190:42441'"
time="2019-11-29T22:07:50+01:00" level=info msg="New client connected: '62.131.70.190:60905'"
time="2019-11-29T22:07:51+01:00" level=info msg="New client connected: '62.131.70.190:34752'"
time="2019-11-29T22:07:52+01:00" level=info msg="New client connected: '62.131.70.190:58098'"
fatal error: concurrent map iteration and map write

goroutine 19 [running]:
runtime.throw(0x8a8bc9, 0x26)
        /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc0000e1828 sp=0xc0000e17f8 pc=0x42f6c2
runtime.mapiternext(0xc0000e1f60)
        /usr/local/go/src/runtime/map.go:858 +0x579 fp=0xc0000e18b0 sp=0xc0000e1828 pc=0x410c49
main.receiveFromWireguard(0xc00009e240, 0xc00009e248)
        /home/travis/build/porech/engarde/cmd/engarde-server/main.go:182 +0x207 fp=0xc0000e1fd0 sp=0xc0000e18b0 pc=0x7b5697
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000e1fd8 sp=0xc0000e1fd0 pc=0x45bf31
created by main.main
        /home/travis/build/porech/engarde/cmd/engarde-server/main.go:124 +0x4c4

goroutine 1 [runnable]:
syscall.Syscall6(0x2d, 0x5, 0xc0000e368c, 0x5dc, 0x0, 0xc0000e3360, 0xc0000e3354, 0x50, 0x5dc, 0x0)
        /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
syscall.recvfrom(0x5, 0xc0000e368c, 0x5dc, 0x5dc, 0x0, 0xc0000e3360, 0xc0000e3354, 0xffffffffffffffff, 0xc0000e3378, 0x42c27a)
        /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1618 +0xa3
syscall.Recvfrom(0x5, 0xc0000e368c, 0x5dc, 0x5dc, 0x0, 0xffffffffffffffff, 0x0, 0x0, 0xd43a80, 0xfc6620)
        /usr/local/go/src/syscall/syscall_unix.go:273 +0xaf
internal/poll.(*FD).ReadFrom(0xc000094280, 0xc0000e368c, 0x5dc, 0x5dc, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:215 +0x13e
net.(*netFD).readFrom(0xc000094280, 0xc0000e368c, 0x5dc, 0x5dc, 0xc0000e3588, 0x5b3abb, 0xc000094200, 0xc0000e368c, 0x50)
        /usr/local/go/src/net/fd_unix.go:208 +0x5b
net.(*UDPConn).readFrom(0xc00009e248, 0xc0000e368c, 0x5dc, 0x5dc, 0x50, 0x0, 0x0, 0xbae54185f3812601)
        /usr/local/go/src/net/udpsock_posix.go:47 +0x6a
net.(*UDPConn).ReadFromUDP(0xc00009e248, 0xc0000e368c, 0x5dc, 0x5dc, 0xc00010b920, 0x50, 0x0, 0x0)
        /usr/local/go/src/net/udpsock.go:109 +0x5d
main.receiveFromClient(0xc00009e248, 0xc00009e240, 0xc00010b920)
        /home/travis/build/porech/engarde/cmd/engarde-server/main.go:138 +0x79
main.main()
        /home/travis/build/porech/engarde/cmd/engarde-server/main.go:125 +0x4ef
engarde-server ver. 996e6b3 (master)
time="2019-11-29T22:13:01+01:00" level=info msg="Listening on 0.0.0.0:59402"
time="2019-11-29T22:13:28+01:00" level=info msg="New client connected: '192.145.56.17:60386'"
time="2019-11-29T22:13:31+01:00" level=info msg="New client connected: '62.131.70.190:35177'"
root@redundant:~#

[ale-rinaldi]

Ouch, that's bad! I'll try to sort it out asap!

[Marctraider]

Cool! I wonder if its somehow all related to why i cannot reliably SQM a single wan interface without the whole tunnel going bonkers?

It feels to me like it currently doesn't like desynced speeds with major ratio difference. Hard to explain.

[ale-rinaldi]

Hello, I spotted the reason of this crash and solved it. It was a minimal change so I put it directly in the master branch. Please download the latest build and you should run just fine ;)

As for the SQM problem, I don't think it's related to the crash: can you please open an issue with some detail about it, and maybe some steps to reproduce? Before that, please try leaving engarde aside and try the same thing on a direct Wireguard tunnel: if you experience the same issue, you'll probably need to open the issue on Wireguard.

Thanks again!

[Marctraider]

Thanks, and will do :-)