search

danielknobe / blobbyvolley2

Official continuation of the famous Blobby Volley 1.x arcade game.
GNU General Public License v2.0
161 stars 21 forks source link

critical server crash since Blobby Volley 1.1 #133

Closed danielknobe closed 1 year ago

danielknobe commented 1 year ago

Surely occurs after a certain time. End of log:

Jun 13 21:53:08 DKBlobbyServer blobby-server[746]: Blobby Server Status Report 844h running 
                                                    packet count: 2594407
                                                    accepted connections: 363
                                                    started games: 14
                                                    game steps: 1300655
Jun 13 22:53:08 DKBlobbyServer blobby-server[746]: Blobby Server Status Report 845h running 
                                                    packet count: 2594407
                                                    accepted connections: 363
                                                    started games: 14
                                                    game steps: 1300655
Jun 13 23:53:09 DKBlobbyServer blobby-server[746]: Blobby Server Status Report 846h running 
                                                    packet count: 2594407
                                                    accepted connections: 363
                                                    started games: 14
                                                    game steps: 1300655
Jun 14 00:14:52 DKBlobbyServer blobby-server[746]: Connection incoming (24) from 35.68.178.45:43946, 1 clients connected now
Jun 14 00:14:53 DKBlobbyServer blobby-server[746]: Connection closed (26) from 35.68.178.45:43946, 0 clients connected now
Jun 14 00:53:10 DKBlobbyServer blobby-server[746]: Blobby Server Status Report 847h running 
                                                    packet count: 2594413
                                                    accepted connections: 364
                                                    started games: 14
                                                    game steps: 1300655
Jun 14 01:53:11 DKBlobbyServer blobby-server[746]: Blobby Server Status Report 848h running 
                                                    packet count: 2594413
                                                    accepted connections: 364
                                                    started games: 14
                                                    game steps: 1300655
Jun 14 02:08:20 DKBlobbyServer blobby-server[746]: Connection incoming (24) from 171.63.255.142:56399, 1 clients connected now
Jun 14 02:08:20 DKBlobbyServer blobby-server[746]: Connection closed (26) from 171.63.255.142:56399, 0 clients connected now
Jun 14 02:16:31 DKBlobbyServer blobby-server[746]: Connection incoming (24) from 171.63.255.142:49389, 1 clients connected now
Jun 14 02:16:31 DKBlobbyServer blobby-server[746]: Connection closed (26) from 171.63.255.142:49389, 0 clients connected now
Jun 14 02:16:40 DKBlobbyServer blobby-server[746]: Connection incoming (24) from 171.63.255.142:51393, 1 clients connected now
Jun 14 02:16:40 DKBlobbyServer blobby-server[746]: Player added (71) from 171.63.255.142:51393 (Right Player), 1 players available
Jun 14 02:17:16 DKBlobbyServer systemd[1]: blobbyvolley2_cpp.service: A process of this unit has been killed by the OOM killer.
Jun 14 02:17:16 DKBlobbyServer systemd[1]: blobbyvolley2_cpp.service: Main process exited, code=killed, status=9/KILL
Jun 14 02:17:16 DKBlobbyServer systemd[1]: blobbyvolley2_cpp.service: Failed with result 'oom-kill'.
Jun 14 02:17:16 DKBlobbyServer systemd[1]: blobbyvolley2_cpp.service: Consumed 1d 10h 43min 59.037s CPU time.

Actually no good idea what can be the reason for that.

ngc92 commented 1 year ago

OOM killer would mean memleak, right?

danielknobe commented 1 year ago

Yes I think so. I restarted the server with memory usage logging. I will report when I have a result.

danielknobe commented 1 year ago

On the second run the error is a bit different (status=11/SEGV):

Jun 17 05:40:59 DKBlobbyServer blobby-server[608407]: Connection incoming (24) from 60.237.226.176:52865, 1 clients connected now
Jun 17 05:40:59 DKBlobbyServer blobby-server[608407]: Connection closed (26) from 60.237.226.176:52865, 0 clients connected now
Jun 17 05:41:03 DKBlobbyServer blobby-server[608407]: Connection incoming (24) from 60.237.226.176:56073, 1 clients connected now
Jun 17 05:41:03 DKBlobbyServer blobby-server[608407]: Connection closed (26) from 60.237.226.176:56073, 0 clients connected now
Jun 17 05:41:04 DKBlobbyServer blobby-server[608407]: Connection incoming (24) from 60.237.226.176:56074, 1 clients connected now
Jun 17 05:41:04 DKBlobbyServer blobby-server[608407]: Player added (71) from 60.237.226.176:56074 (Left Player), 1 players available
Jun 17 05:41:33 DKBlobbyServer blobby-server[608407]: Connection incoming (24) from 140.20.206.5:53538, 2 clients connected now
Jun 17 05:41:33 DKBlobbyServer blobby-server[608407]: Connection closed (26) from 140.20.206.5:53538, 1 clients connected now
Jun 17 05:41:34 DKBlobbyServer blobby-server[608407]: Connection incoming (24) from 140.20.206.5:54942, 2 clients connected now
Jun 17 05:41:34 DKBlobbyServer blobby-server[608407]: Player added (71) from 140.20.206.5:54942 (Combat Cock), 2 players available
Jun 17 05:41:45 DKBlobbyServer systemd[1]: blobbyvolley2_cpp.service: Main process exited, code=killed, status=11/SEGV
Jun 17 05:41:45 DKBlobbyServer systemd[1]: blobbyvolley2_cpp.service: Failed with result 'signal'.
Jun 17 05:41:45 DKBlobbyServer systemd[1]: blobbyvolley2_cpp.service: Consumed 1h 29min 35.154s CPU time.

The logged memory was ok in case of this log.

I rerun with core dump enabled.

danielknobe commented 1 year ago

I can reproduce the error now. To do this, start the dedicated server and start two games on it (once 160% and once 100%). After about 30 seconds the server crashes. It seems that this is a race condition. Used datastructures seem not to be thread safe in our scenario of threads (multi writer).

Using a mutex to make bufferedCommands.WriteLock() thread-safe solves the concrete problem. Do you have a good solution or a good data structure to use @ngc92 ? Or should we just lock it?

ngc92 commented 1 year ago

I don't think just using a mutex in WriteLock actually solves the problem, though it might make the symptoms go away here. The underlying race condition is that we access RakServer from multiple threads, even though the class is not designed for that. We'd have to validate that Send can be called in parallel to any other function we might call from the main thread, which includes e.g., making sure that ValidSendTarget doesn't read while the main thread is writing (i.e., one game tries to send a packet, while the main server thread is just adding a new system).

One approach could be to wrap the entire RakServer with a mutex, to ensure that only one thread ever touches it at the same time. Since all our threads are potential readers and writers at some point, I think this is the best we can do with a simple solution.

An alternative would be to buffer each game's packets in its own thread, and handle all communication with RakNet from the main thread. This would remove contentious locking, but introduce additional latency into the pipeline. At the very least, we'd have to remove the throttling in the main thread.

Btw, while looking into this problem, I think I found another race condition: When starting the game, we currently create the new game with its thread, and then remove the two players from the lobby. This only works if the first packets send out from the game thread are earlier than the removal packets from the lobby thread. Otherwise, the client crashes, because we trigger an assert.

danielknobe commented 1 year ago

You are right. To be sure that we are not missing something, it is better to make sure only one thread touches the class the same time at all. Keep it simple.

We should fix the other issue, too. It will probably not have been the last problem in the netcode.

danielknobe commented 1 year ago

I deployed this fix to our gameserver for testing.

danielknobe commented 1 year ago

The gameserver is still crashing :fearful: . Seems the patch doesn't solves the problem completely. Latest log:

Jun 22 20:23:22 DKBlobbyServer blobby-server[729]: loaded rules Crazy Volley - Blitz by chameleon from blitz.lua
Jun 22 20:23:22 DKBlobbyServer blobby-server[729]: loaded rules Crazy Volley - Firewall by chameleon from firewall.lua
Jun 22 20:23:22 DKBlobbyServer blobby-server[729]: loaded rules Crazy Volley - Sticky Mode by chameleon from sticky_mode.lua
Jun 22 20:23:22 DKBlobbyServer blobby-server[729]: Blobby Volley 2 dedicated server version 0.106 started
Jun 22 20:23:22 DKBlobbyServer blobby-server[729]: loaded rules Crazy Volley - Jumping Jack by chameleon from jumping_jack.lua
Jun 22 20:23:22 DKBlobbyServer blobby-server[729]: loaded rules Crazy Volley - Tennis by chameleon from tennis.lua
Jun 22 20:27:40 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 67.9.245.89:42733, 1 clients connected now
Jun 22 20:27:41 DKBlobbyServer blobby-server[729]: Connection closed (26) from 67.9.245.89:42733, 0 clients connected now
Jun 22 20:28:02 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 67.9.245.89:52219, 1 clients connected now
Jun 22 20:28:02 DKBlobbyServer blobby-server[729]: Connection closed (26) from 67.9.245.89:52219, 0 clients connected now
Jun 22 20:28:03 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 67.9.245.89:45784, 1 clients connected now
Jun 22 20:28:03 DKBlobbyServer blobby-server[729]: Player added (71) from 67.9.245.89:45784 (Right Player), 1 players available
Jun 22 20:28:11 DKBlobbyServer blobby-server[729]: Player removed (26) from 67.9.245.89:45784 (Right Player), 0 players available
Jun 22 20:28:11 DKBlobbyServer blobby-server[729]: Connection closed (26) from 67.9.245.89:45784, 0 clients connected now
Jun 22 21:23:22 DKBlobbyServer blobby-server[729]: Blobby Server Status Report 1h running 
                                                    packet count: 22
                                                    accepted connections: 3
                                                    started games: 0
                                                    game steps: 0
Jun 22 22:23:23 DKBlobbyServer blobby-server[729]: Blobby Server Status Report 2h running 
                                                    packet count: 22
                                                    accepted connections: 3
                                                    started games: 0
                                                    game steps: 0
Jun 22 23:23:24 DKBlobbyServer blobby-server[729]: Blobby Server Status Report 3h running 
                                                    packet count: 22
                                                    accepted connections: 3
                                                    started games: 0
                                                    game steps: 0
Jun 23 00:23:25 DKBlobbyServer blobby-server[729]: Blobby Server Status Report 4h running 
                                                    packet count: 22
                                                    accepted connections: 3
                                                    started games: 0
                                                    game steps: 0
Jun 23 01:23:25 DKBlobbyServer blobby-server[729]: Blobby Server Status Report 5h running 
                                                    packet count: 22
                                                    accepted connections: 3
                                                    started games: 0
                                                    game steps: 0
Jun 23 02:23:26 DKBlobbyServer blobby-server[729]: Blobby Server Status Report 6h running 
                                                    packet count: 22
                                                    accepted connections: 3
                                                    started games: 0
                                                    game steps: 0
Jun 23 03:23:27 DKBlobbyServer blobby-server[729]: Blobby Server Status Report 7h running 
                                                    packet count: 22
                                                    accepted connections: 3
                                                    started games: 0
                                                    game steps: 0
Jun 23 04:23:27 DKBlobbyServer blobby-server[729]: Blobby Server Status Report 8h running 
                                                    packet count: 22
                                                    accepted connections: 3
                                                    started games: 0
                                                    game steps: 0
Jun 23 05:23:28 DKBlobbyServer blobby-server[729]: Blobby Server Status Report 9h running 
                                                    packet count: 22
                                                    accepted connections: 3
                                                    started games: 0
                                                    game steps: 0
Jun 23 06:03:53 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:56988, 1 clients connected now
Jun 23 06:03:53 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:56988, 0 clients connected now
Jun 23 06:04:09 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:49555, 1 clients connected now
Jun 23 06:04:09 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:49555, 0 clients connected now
Jun 23 06:04:13 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:57219, 1 clients connected now
Jun 23 06:04:13 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:57219, 0 clients connected now
Jun 23 06:04:18 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:57220, 1 clients connected now
Jun 23 06:04:18 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:57220, 0 clients connected now
Jun 23 06:04:19 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:57221, 1 clients connected now
Jun 23 06:04:19 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:57221, 0 clients connected now
Jun 23 06:04:22 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:57222, 1 clients connected now
Jun 23 06:04:22 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:57222 (Right Player), 1 players available
Jun 23 06:04:47 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:63490, 2 clients connected now
Jun 23 06:04:48 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:63490, 1 clients connected now
Jun 23 06:04:48 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:63491, 2 clients connected now
Jun 23 06:04:48 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:63491 (Right Player), 2 players available
Jun 23 06:04:53 DKBlobbyServer blobby-server[729]: loaded rules BV2 Default Rules by Blobby Volley 2 Developers from default.lua
Jun 23 06:04:53 DKBlobbyServer blobby-server[729]: Created game 'Right Player' vs. 'Right Player', rules: 'default.lua'
Jun 23 06:06:14 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 171.63.255.142:38269, 3 clients connected now
Jun 23 06:06:14 DKBlobbyServer blobby-server[729]: Connection closed (26) from 171.63.255.142:38269, 2 clients connected now
Jun 23 06:06:25 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 171.63.255.142:35387, 3 clients connected now
Jun 23 06:06:25 DKBlobbyServer blobby-server[729]: Connection closed (26) from 171.63.255.142:35387, 2 clients connected now
Jun 23 06:06:28 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 171.63.255.142:34890, 3 clients connected now
Jun 23 06:06:29 DKBlobbyServer blobby-server[729]: Connection closed (26) from 171.63.255.142:34890, 2 clients connected now
Jun 23 06:06:30 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 171.63.255.142:51915, 3 clients connected now
Jun 23 06:06:30 DKBlobbyServer blobby-server[729]: Connection closed (26) from 171.63.255.142:51915, 2 clients connected now
Jun 23 06:06:49 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 171.63.255.142:50335, 3 clients connected now
Jun 23 06:06:50 DKBlobbyServer blobby-server[729]: Connection closed (26) from 171.63.255.142:50335, 2 clients connected now
Jun 23 06:06:50 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 171.63.255.142:50684, 3 clients connected now
Jun 23 06:06:51 DKBlobbyServer blobby-server[729]: Connection closed (26) from 171.63.255.142:50684, 2 clients connected now
Jun 23 06:07:45 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:57222 (Right Player), 1 players available
Jun 23 06:07:45 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:57222, 1 clients connected now
Jun 23 06:07:45 DKBlobbyServer blobby-server[729]: Removed game 250.22.161.147:63491 vs 250.22.161.147:57222 from gamelist
Jun 23 06:07:46 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:56305, 2 clients connected now
Jun 23 06:07:46 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:56305, 1 clients connected now
Jun 23 06:07:48 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:63491 (Right Player), 0 players available
Jun 23 06:07:48 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:63491, 0 clients connected now
Jun 23 06:07:49 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:55201, 1 clients connected now
Jun 23 06:07:49 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:55201, 0 clients connected now
Jun 23 06:07:51 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:57250, 1 clients connected now
Jun 23 06:07:51 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:57250 (Right Player), 1 players available
Jun 23 06:08:12 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:59518, 2 clients connected now
Jun 23 06:08:12 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:59518, 1 clients connected now
Jun 23 06:08:13 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:59519, 2 clients connected now
Jun 23 06:08:13 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:59519 (Right Player), 2 players available
Jun 23 06:08:16 DKBlobbyServer blobby-server[729]: loaded rules BV2 Default Rules by Blobby Volley 2 Developers from default.lua
Jun 23 06:08:16 DKBlobbyServer blobby-server[729]: Created game 'Right Player' vs. 'Right Player', rules: 'default.lua'
Jun 23 06:10:02 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:59519 (Right Player), 1 players available
Jun 23 06:10:02 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:59519, 1 clients connected now
Jun 23 06:10:02 DKBlobbyServer blobby-server[729]: Removed game 250.22.161.147:59519 vs 250.22.161.147:57250 from gamelist
Jun 23 06:10:12 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:57250 (Right Player), 0 players available
Jun 23 06:10:12 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:57250, 0 clients connected now
Jun 23 06:10:13 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:63148, 1 clients connected now
Jun 23 06:10:13 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:63148, 0 clients connected now
Jun 23 06:10:15 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:63149, 1 clients connected now
Jun 23 06:10:15 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:63149, 0 clients connected now
Jun 23 06:10:17 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:63150, 1 clients connected now
Jun 23 06:10:17 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:63150 (Right Player), 1 players available
Jun 23 06:10:33 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:53665, 2 clients connected now
Jun 23 06:10:33 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:53665, 1 clients connected now
Jun 23 06:10:34 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:53666, 2 clients connected now
Jun 23 06:10:34 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:53666 (Right Player), 2 players available
Jun 23 06:10:37 DKBlobbyServer blobby-server[729]: loaded rules BV2 Default Rules by Blobby Volley 2 Developers from default.lua
Jun 23 06:10:37 DKBlobbyServer blobby-server[729]: Created game 'Right Player' vs. 'Right Player', rules: 'default.lua'
Jun 23 06:13:06 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:53666 (Right Player), 1 players available
Jun 23 06:13:06 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:53666, 1 clients connected now
Jun 23 06:13:06 DKBlobbyServer blobby-server[729]: Removed game 250.22.161.147:53666 vs 250.22.161.147:63150 from gamelist
Jun 23 06:13:06 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:63150 (Right Player), 0 players available
Jun 23 06:13:06 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:63150, 0 clients connected now
Jun 23 06:13:18 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:56818, 1 clients connected now
Jun 23 06:13:18 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:56818, 0 clients connected now
Jun 23 06:13:19 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:56819, 1 clients connected now
Jun 23 06:13:19 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:56819 (Right Player), 1 players available
Jun 23 06:14:21 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:54439, 2 clients connected now
Jun 23 06:14:22 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:54439, 1 clients connected now
Jun 23 06:14:22 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:54440, 2 clients connected now
Jun 23 06:14:22 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:54440 (Theo), 2 players available
Jun 23 06:14:35 DKBlobbyServer blobby-server[729]: loaded rules BV2 Default Rules by Blobby Volley 2 Developers from default.lua
Jun 23 06:14:35 DKBlobbyServer blobby-server[729]: Created game 'Theo' vs. 'Right Player', rules: 'default.lua'
Jun 23 06:17:07 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:56819 (Right Player), 1 players available
Jun 23 06:17:07 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:56819, 1 clients connected now
Jun 23 06:17:07 DKBlobbyServer blobby-server[729]: Removed game 250.22.161.147:54440 vs 250.22.161.147:56819 from gamelist
Jun 23 06:17:09 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:54440 (Theo), 0 players available
Jun 23 06:17:09 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:54440, 0 clients connected now
Jun 23 06:17:21 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:62372, 1 clients connected now
Jun 23 06:17:21 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:62372, 0 clients connected now
Jun 23 06:18:09 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:60452, 1 clients connected now
Jun 23 06:18:09 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:60452, 0 clients connected now
Jun 23 06:18:10 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:55253, 1 clients connected now
Jun 23 06:18:10 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:55253 (Rick), 1 players available
Jun 23 06:18:22 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:61336, 2 clients connected now
Jun 23 06:18:22 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:61336, 1 clients connected now
Jun 23 06:18:23 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:61337, 2 clients connected now
Jun 23 06:18:23 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:61337 (Theo), 2 players available
Jun 23 06:18:26 DKBlobbyServer blobby-server[729]: loaded rules BV2 Default Rules by Blobby Volley 2 Developers from default.lua
Jun 23 06:18:26 DKBlobbyServer blobby-server[729]: Created game 'Theo' vs. 'Rick', rules: 'default.lua'
Jun 23 06:22:12 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:61337 (Theo), 1 players available
Jun 23 06:22:12 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:61337, 1 clients connected now
Jun 23 06:22:12 DKBlobbyServer blobby-server[729]: Removed game 250.22.161.147:61337 vs 250.22.161.147:55253 from gamelist
Jun 23 06:23:20 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:55253 (Rick), 0 players available
Jun 23 06:23:20 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:55253, 0 clients connected now
Jun 23 06:23:22 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:62386, 1 clients connected now
Jun 23 06:23:22 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:62386, 0 clients connected now
Jun 23 06:23:23 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:62387, 1 clients connected now
Jun 23 06:23:23 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:62387 (Rick), 1 players available
Jun 23 06:23:29 DKBlobbyServer blobby-server[729]: Blobby Server Status Report 10h running 
                                                    packet count: 117782
                                                    accepted connections: 38
                                                    started games: 5
                                                    game steps: 60345
Jun 23 06:24:02 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:52004, 2 clients connected now
Jun 23 06:24:02 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:52004, 1 clients connected now
Jun 23 06:24:14 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:64295, 2 clients connected now
Jun 23 06:24:14 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:64295, 1 clients connected now
Jun 23 06:24:20 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:64098, 2 clients connected now
Jun 23 06:24:20 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:64098 (Louis), 2 players available
Jun 23 06:24:24 DKBlobbyServer blobby-server[729]: loaded rules BV2 Default Rules by Blobby Volley 2 Developers from default.lua
Jun 23 06:24:24 DKBlobbyServer blobby-server[729]: Created game 'Louis' vs. 'Rick', rules: 'default.lua'
Jun 23 06:26:28 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:64098 (Louis), 1 players available
Jun 23 06:26:28 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:64098, 1 clients connected now
Jun 23 06:26:28 DKBlobbyServer blobby-server[729]: Removed game 250.22.161.147:64098 vs 250.22.161.147:62387 from gamelist
Jun 23 06:26:28 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:62387 (Rick), 0 players available
Jun 23 06:26:28 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:62387, 0 clients connected now
Jun 23 06:26:52 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:64818, 1 clients connected now
Jun 23 06:26:53 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:64818, 0 clients connected now
Jun 23 06:26:53 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:55150, 1 clients connected now
Jun 23 06:26:53 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:55150 (Louis), 1 players available
Jun 23 06:27:00 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:63020, 2 clients connected now
Jun 23 06:27:00 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:63020, 1 clients connected now
Jun 23 06:27:05 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:50985, 2 clients connected now
Jun 23 06:27:05 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:50985 (Theo), 2 players available
Jun 23 06:27:58 DKBlobbyServer blobby-server[729]: loaded rules BV2 Default Rules by Blobby Volley 2 Developers from default.lua
Jun 23 06:27:58 DKBlobbyServer blobby-server[729]: Created game 'Louis' vs. 'Theo', rules: 'default.lua'
Jun 23 06:30:49 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:50985 (Theo), 1 players available
Jun 23 06:30:49 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:50985, 1 clients connected now
Jun 23 06:30:49 DKBlobbyServer blobby-server[729]: Removed game 250.22.161.147:55150 vs 250.22.161.147:50985 from gamelist
Jun 23 06:31:27 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:55150 (Louis), 0 players available
Jun 23 06:31:27 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:55150, 0 clients connected now
Jun 23 06:31:40 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:59207, 1 clients connected now
Jun 23 06:31:40 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:59207, 0 clients connected now
Jun 23 06:31:41 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:60172, 1 clients connected now
Jun 23 06:31:41 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:60172 (Rick), 1 players available
Jun 23 06:32:01 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:54547, 2 clients connected now
Jun 23 06:32:01 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:54547, 1 clients connected now
Jun 23 06:32:01 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:54548, 2 clients connected now
Jun 23 06:32:01 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:54548 (Theo), 2 players available
Jun 23 06:32:04 DKBlobbyServer blobby-server[729]: loaded rules BV2 Default Rules by Blobby Volley 2 Developers from default.lua
Jun 23 06:32:04 DKBlobbyServer blobby-server[729]: Created game 'Theo' vs. 'Rick', rules: 'default.lua'
Jun 23 06:34:36 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:54548 (Theo), 1 players available
Jun 23 06:34:36 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:54548, 1 clients connected now
Jun 23 06:34:36 DKBlobbyServer blobby-server[729]: Removed game 250.22.161.147:54548 vs 250.22.161.147:60172 from gamelist
Jun 23 06:34:37 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:60172 (Rick), 0 players available
Jun 23 06:34:37 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:60172, 0 clients connected now
Jun 23 06:34:47 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:53138, 1 clients connected now
Jun 23 06:34:47 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:53138, 0 clients connected now
Jun 23 06:34:48 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:58543, 1 clients connected now
Jun 23 06:34:48 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:58543 (Louis), 1 players available
Jun 23 06:35:27 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:54949, 2 clients connected now
Jun 23 06:35:28 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:54949, 1 clients connected now
Jun 23 06:35:30 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:56738, 2 clients connected now
Jun 23 06:35:30 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:56738 (Rick), 2 players available
Jun 23 06:35:51 DKBlobbyServer blobby-server[729]: loaded rules BV2 Default Rules by Blobby Volley 2 Developers from default.lua
Jun 23 06:35:51 DKBlobbyServer blobby-server[729]: Created game 'Rick' vs. 'Louis', rules: 'default.lua'
Jun 23 06:41:30 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:56738 (Rick), 1 players available
Jun 23 06:41:30 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:56738, 1 clients connected now
Jun 23 06:41:30 DKBlobbyServer blobby-server[729]: Removed game 250.22.161.147:56738 vs 250.22.161.147:58543 from gamelist
Jun 23 06:42:18 DKBlobbyServer blobby-server[729]: Player removed (26) from 250.22.161.147:58543 (Louis), 0 players available
Jun 23 06:42:18 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:58543, 0 clients connected now
Jun 23 06:42:21 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:49311, 1 clients connected now
Jun 23 06:42:21 DKBlobbyServer blobby-server[729]: Connection closed (26) from 250.22.161.147:49311, 0 clients connected now
Jun 23 06:42:25 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 250.22.161.147:49362, 1 clients connected now
Jun 23 06:42:25 DKBlobbyServer blobby-server[729]: Player added (71) from 250.22.161.147:49362 (Theo), 1 players available
Jun 23 06:53:37 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 65.220.28.103:64278, 2 clients connected now
Jun 23 06:53:37 DKBlobbyServer blobby-server[729]: Connection closed (26) from 65.220.28.103:64278, 1 clients connected now
Jun 23 06:53:44 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 65.220.28.103:50027, 2 clients connected now
Jun 23 06:53:44 DKBlobbyServer blobby-server[729]: Connection closed (26) from 65.220.28.103:50027, 1 clients connected now
Jun 23 06:53:48 DKBlobbyServer blobby-server[729]: Connection incoming (24) from 65.220.28.103:58293, 2 clients connected now
Jun 23 06:53:48 DKBlobbyServer blobby-server[729]: Player added (71) from 65.220.28.103:58293 (Right Player), 2 players available
Jun 23 06:54:09 DKBlobbyServer systemd[1]: blobbyvolley2_cpp.service: Main process exited, code=killed, status=11/SEGV
Jun 23 06:54:09 DKBlobbyServer systemd[1]: blobbyvolley2_cpp.service: Failed with result 'signal'.
Jun 23 06:54:09 DKBlobbyServer systemd[1]: blobbyvolley2_cpp.service: Consumed 25min 15.968s CPU time.
ngc92 commented 1 year ago

@danielknobe Do you have a backtrace?

danielknobe commented 1 year ago

@ngc92 currently not because systemd-tmpfiles was faster than me. I increased the expiration time for coredumps and restarted the server.

danielknobe commented 1 year ago

Found a invalid iter access using valgrind. I deployed this fix to our server.