Cyclic error showing in game log, repeated sim drops and network activity lapses - Githubissues

jpcsupplies / Economy_mod

Basic Economy System for Space Engineers

13 stars 12 forks source link

Cyclic error showing in game log, repeated sim drops and network activity lapses #179

Open Tharatan opened 6 years ago

Tharatan commented 6 years ago

We're seeing repeated instances on our dedicated server (Hardcore Survivalists) of sim dropping to 0.02, which co-incide with the server sending zero data in cycles. A few of our more enterprising players decided to see if it was affecting anyone else, and found the same pattern of sim drops on other servers running Frontier Economy right now (for example, Unknown Realm).

We also see constant fields of the following in the game logs: 18:32:37 GameAnalyticsSDK.Net.Logging.GALogger Warning/GameAnalytics: Event queue: Failed to send events. 18:32:46 GameAnalyticsSDK.Net.Logging.GALogger Warning/GameAnalytics: Event queue: Failed to send events. 18:32:55 GameAnalyticsSDK.Net.Logging.GALogger Warning/GameAnalytics: Event queue: Failed to send events.

Is there a cyclical command in the script firing here that could be causing the problem?

https://cdn.discordapp.com/attachments/384926717262823446/410954652855697408/20180207180121_1.jpg

https://cdn.discordapp.com/attachments/384926717262823446/410954803376553984/20180207180916_1.jpg

https://cdn.discordapp.com/attachments/384926717262823446/410955054208516114/20180207181858_1.jpg

midspace commented 6 years ago

There are a number of timed events.

Trade timeouts are checked every 10 seconds. These are Player to Player trades. This is done server side.
DelayedConnectionRequest occurs every 10 seconds on the Client side, until it receives initial configuration information from the server, then is stops.
LCD updates. These occur at the minimum of every 100 game ticks. This can be adjusted to be slower via /econfig lcdDisplayInterval 10 (in seconds) or turned off via /econfig enablelcds off

At the moment, I don't have enough information to draw any sort of conclusion. Which log file are the GameAnalytics messages appearing in? The Server or the Client?

midspace commented 6 years ago

I have just checking the LCD on a server, it looks like it may not be obeying the configuration. I recently re-wrote the code to improve performance, but it looks like I may have done something wrong.

midspace commented 6 years ago

Sigh. I was looking at the wrong LCDs on the server. I can't detect any issues, at least on this server.

midspace commented 6 years ago

This is the network graph from our public server. 20180208191314_1

Understandably, the data is a little more busy when close to grids.

midspace commented 6 years ago

I'm remembering now, that GameAnalytics is Keen's data Analytics, which collects information and sends to the GameAnalytics servers for them to analyse player behaviour. (See what is popular and what isn't).

If you are getting that message, then it's most likely the server cannot connect to the GameAnalytics server for some reason. It isn't necessarily an issue on your end. The DS I'm running here is suddenly showing that message, which is unusual.

Tharatan commented 6 years ago

Not sure if it helps, but here's the Data file for our instance. Anything else that might help narrow things down? If its not the mod and its just KSWH having caused something odd, then my apologies - just wanted to follow up based on the similarities noted between Economy-using and non-Economy DS's.

EconomyData.zip

midspace commented 6 years ago

I'd be more interested in seeing one of these servers both with and without the economy mod.

Tharatan commented 6 years ago

Our server discord is at https://discord.gg/tTD3ynJ - what time/day would be good for you?

jpcsupplies commented 6 years ago

This is complete speculation on my part- but Analytic's would be making socket connections, it is possible that also log jams our client/server communication when it waits for the analytic connection to time out.. i seem to recall some similar behavior in the old VB when using sockets making the program appear to hang until the open socket gave up and closed. In that case it didn't matter if the connection failed the program couldnt do its thing so you didnt really notice. In SE however its multi tasking and threading a 100 other things at the same time, which would probably grind to a halt if its queued on the same thread as the deadlocked socket waiting to timeout. @midspace does poking around the analytic code in the SE game source offer any clues as to how it is threading its socket connections? is it on the same thread as our communication uses in modapi by any amazing coincidence? I suppose an indirect way to test the theory is find another mod making a lot of use of client/server coms and see if the same effect occurs.

Tharatan commented 6 years ago

Just a thought - this is on a DS, and they haven't allowed/enabled multithreading on DS's to the best of my knowledge.