Server Crash after failed request to update ELO

[theCrius]

Hi, apologies if this is not the right place to write about this or if it's misconfiguration on my part. I tried posting on the discord but received no response.

This evening the Hub I host for my community crashed and this is the extract of the logs:

[2024.01.31-21.02.49:529][249]LogExit: IpNetDriver_13 IpNetDriver_13 shut down
[2024.01.31-21.02.49:529][249]LogInit: BSD: Socket queue 65536 / 65536
[2024.01.31-21.08.11:912][992]LogHTTP:Warning: 0x7fa8717bdf00: request failed, libcurl error: 0 (No error)
[2024.01.31-21.08.11:912][992]LogHTTP:Warning: 0x7fa8717b0e80: request failed, libcurl error: 0 (No error)
[2024.01.31-21.08.11:912][992]LogHTTP:Warning: 0x7fa8717b6280: request failed, libcurl error: 0 (No error)
[2024.01.31-21.11.28:581][523]LogGameStats:Warning: Stats write failed https://master-ut4.timiimit.com/ut/api/stats/accountId/[REMOVED]/bulk?ownertype=1 {"type":"https://tools.ietf.org/html/rfc7235#section-3.1","title":"Unauthorized","status":401,"traceId":"00-f8655927a09cca2cddf7c09a02134439-042c9c0771703dd8-00"}
[2024.01.31-21.11.28:638][529]UT:Warning: Failed to report Match Results to the server. User ELOs will not be updated. (0)
[2024.01.31-21.11.28:638][529]LogOnline:Error: OSS: McpUtils request https://master-ut4.timiimit.com/ut/api/game/v2/ratings/team/match_result failed. code=401 response={"errorCode":null,"errorMessage":"Only trusted servers may update ELO","messageVars":[],"numericErrorCode":0,"originatingService":null,"intent":null,"error_description":"Only trusted servers may update ELO","error":null}
[2024.01.31-21.11.28:840][547]LogOnline:Warning: MCP: Invalid response. CorrId=UE4-618318291C01151F000B02115284F41C code=200 errorcode=InvalidJson errormessage=Unable to parse server response. Bad JSON. Error Null Stream Line: 1 Ch: 0 errorraw=
[2024.01.31-21.11.28:840][547]LogOnline:Warning: MCP: WriteUserFile request failed. Unable to parse server response. Bad JSON. Error Null Stream Line: 1 Ch: 0
[2024.01.31-21.11.28:840][547]LogOnline:Warning: MCP: Invalid response. CorrId=UE4-6182E9C01C01151F000B02105284E8CB code=200 errorcode=InvalidJson errormessage=Unable to parse server response. Bad JSON. Error Null Stream Line: 1 Ch: 0 errorraw=
[2024.01.31-21.11.28:840][547]LogOnline:Warning: MCP: WriteUserFile request failed. Unable to parse server response. Bad JSON. Error Null Stream Line: 1 Ch: 0
[2024.01.31-21.11.28:974][559]LogOnline:Warning: MCP: Invalid response. CorrId=UE4-61841EF01C01151F000B02135284FDD3 code=200 errorcode=InvalidJson errormessage=Unable to parse server response. Bad JSON. Error Null Stream Line: 1 Ch: 0 errorraw=
[2024.01.31-21.11.28:974][559]LogOnline:Warning: MCP: WriteUserFile request failed. Unable to parse server response. Bad JSON. Error Null Stream Line: 1 Ch: 0
[2024.01.31-21.13.03:533][987]LogUObjectGlobals: BeginLoad(/Game/EpicInternal/Teams/ThunderCrash/CosmeticItems/TC_HockeyMask/BP_TC_HockeyMask) is flushing async loading
[2024.01.31-21.13.09:539][523]LogEngine: Server switch level: /Game/RestrictedAssets/Maps/WIP/DM-Cannon?NextMap=1

A user reported trying to run an Elimination match and the server crashed shorty after finishing it, but it could be unrelated as there were other matches going on at the time.

Unfortunately I wasn't online so I can't provide much more details.

Feel free to ask for any additional information I might provide. This is the repo I use: https://github.com/theCrius/ut4hub

Thank you

[timiimit]

This looks like just a random crash because the line with Server switch level already happened after the errors in the log. Game is pretty buggy sometimes and it is generally recommend to have some checker restart the hub if it crashes. Hub usually even stops itself after 4 hours of runtime or so if I'm not mistaken.

https://master-ut4.timiimit.com/ut/api/game/v2/ratings/team/match_result request fails because your hub does not have trusted status.

https://master-ut4.timiimit.com/ut/api/stats/accountId/[REMOVED]/bulk?ownertype=1 fails because UT4's server sends requests out of order and master server expects proper ordering. This is something that needs to be looked into and probably have ordering restrictions removed in the future.

Line pairs of WriteUserFile request failed and Invalid response are related to the same request (https://master-ut4.timiimit.com/ut/api/cloudstorage/user/<AccountID>/stats.json). The problem is either a master server bug or master server expects proper ordering like with the request above. Need to further investigate for this one. In any case it should be fixed.

Since this issue was made specifically about the crash, i think this issue can be closed unless you have further questions. Additionally, it would be good if 2 new issues were opened regarding each of the aforementioned master server bugs.

[theCrius]

Is there anything I can do as server admin to prevent this?

The server is already staying up only 4 hours (20.30-00.30 local time for the player).

[timiimit]

Not really. When it comes to UT4 and crashing, its just something you have to deal with every once in a while. Doing a check and restarting is the best that can be done.

[theCrius]

Alright, thank you. I'll see if I can mess with docker so that it can intercept these errors and force a restart of the container when it happens.