Server CPU %100 lagging with 100 online

[temel61]

I have a ubuntu 20.0 32 gb ram and 2 cpu server in microsoft azure.I have around 100 online and 3 maps.My cpu usage is around %95-100.Where is the mistake ?

FPS =+30 Ping = 100-150 ms

Monster = 1000 Harvestables = 100

How can I reduce my cpu usage ? .32 GB RAM is not enough for 100 online ?.Should I increase my RAM to 128 GB ? Or need sometings else?

[insthync]

I am going to change the network updating workflow, but I think you should also check other scene components too.

[temel61]

What do you prefer for me now?. Which settings should I do to reduce cpu usage?.Optimization ? network packet send interval?

[insthync]

Can you use Profiler to find which components use a lots of CPU?, may run an offline game and move to some scene to check CPU usage

[temel61]

of course,I will do that for you today.Should I increase my server RAM?. 32 GB RAM ie enough for 100 online players ?

[insthync]

It should be enough, I guess that what uses a lots of CPU is the character's animation because it is always updating, I will make it stop updating when there are no subscribing users

[temel61]

My server Performance Report

https://drive.google.com/file/d/1yRC9OWRsJSlpO8mJip-Y5Tg9VO1a3PnJ/view?usp=share_link

https://drive.google.com/file/d/18FASU9O0rtkSaIW1k7YhSU_tZokWwZYm/view?usp=share_link

[temel61]

[sdk2016]

looks like the same map was created multiple times. I have experienced something like this. I believe it is related to the problem as in this link: https://github.com/suriyun-production/mmorpg-kit-docs/issues/2007

[temel61]

I am using this command -> nohup ./Build.x86_64 -startCentralServer -startMapSpawnServer -startDatabaseServer > /dev/null 2>&1&

[sdk2016]

a map that is created repeatedly due to timeouts which causes very high server resource usage

[temel61]

how can I solve it ?

[temel61]

where is mistake ?

[temel61]

[temel61]

{ "databaseOptionIndex" : 0,

"centralAddress" : "127.0.0.1", "centralPort" : 7000, "clusterPort" : 7010, "centralMaxConnections" : 10000, "machineAddress" : "98.71.130.88",

"mapSpawnPort" : 6001, "mapSpawnMaxConnections" : 10000, "spawnExePath" : "/root/Build.x86_64", "notSpawnInBatchMode" : false, "spawnStartPort" : 8000, "spawnMaps" : ["Town","Village","MonkeyDungeon"],

"databaseManagerAddress" : "localhost", "databaseManagerPort" : 6003,

"chatPort" : 3002, "chatMaxConnections" : 1000 }

[sdk2016]

you can find out if there is a timeout by reading the file in the log folder, on the server. I don't know what causes the timeout. I'm back to the old version at the moment.

[temel61]

I am using v1.78 kit, ubuntu 20.0

[temel61]

Log_Database_Central_MapSpawn.log

a lot of database error :(

[sdk2016]

version 1.78f has fix several issue:

 Try fix storage items race condition issues.
Try fix mail rewarding race condition issues.

i think your game is having storage race condition which causes duplicate item id's

[temel61]

I has changed core kit, I need to find required code changes and try to fix it

[temel61]

I will remove human race in my game, I hope this works.

[insthync]

What is that human race?

[temel61]

I added human race for all classes, I guess this cause database error

[insthync]

I don't think so.

[temel61]

Try fix storage items race condition issues. Try fix mail rewarding race condition issues.

Wht does race condition mean ?. Which code should I change?

[insthync]

You may try googling "coding race condition" https://learn.microsoft.com/en-us/troubleshoot/developer/visualstudio/visual-basic/language-compilers/race-conditions-deadlocks

I can't remember what I've made changes, but I think it won't be hard to update, some git clients have a file comparing system (I use Smartgit), if you can remember what you've made changes then you can just compare those files before making a commit.

[temel61]

I will remove human race in my game, I hope this works.

it does not work for me

[insthync]

Yep, I know it won't work.

As I saw your images, you are not run 3 map-servers, it is around 16.

You may try to kill all processes before running the servers, you may use the command killall Build.x86_64 before start servers

[insthync]

You also should update, at least update MMO part of commits before the end of FEB, because after that, it is having lots of changes

https://github.com/insthync/UnityMultiplayerARPG_MMO/tree/3d228049da62ac172ebf1da58169d9a6823f2692

[temel61]

Woow.I will try it now.I also use reboot command while running game in server.Maybe this cause a error ?

[temel61]

[insthync]

Is it better?

[temel61]

I reboot server and check process.Started command in my server.Still so much map servers , probably 8 or 16 each map. I am using 1.78c

[temel61]

{ "databaseOptionIndex" : 0,

"centralAddress" : "127.0.0.1", "centralPort" : port, "clusterPort" : 7010, "centralMaxConnections" : 1000, "machineAddress" : "ip",

"mapSpawnPort" : 6001, "mapSpawnMaxConnections" : 1000, "spawnExePath" : "./Build.x86_64", "notSpawnInBatchMode" : false, "spawnStartPort" : 8000, "spawnMaps" : ["City", "Village", "MonkeyDungeon"],

"databaseManagerAddress" : "localhost", "databaseManagerPort" : 6003,

"chatPort" : 3002, "chatMaxConnections" : 1000 }

[insthync]

I won't back to the old version to fix your issues because if it cause in the past then it may be fixed, I think you should include all log files, I don't get why processes started like that. And please show CPU usage when there it was just 3 map-servers?

[temel61]

Log_Database_Central_MapSpawn.log

my 4 log files has 280 mb size for 3 days.It download log files very very slow when I try to download.My server is crying right now with 100 online. I have also often navmesh errors.

[temel61]

My game here,You can try to understand -> https://drive.google.com/file/d/1z6TCEaJIwqfAG95Q1hF7lo6Oyn5Joc-t/view?usp=share_link

[insthync]

How I can understand if I cannot find any error?, give me that client is not helpful, I am not going to find bugs from client.

[temel61]

https://drive.google.com/file/d/1bUHgRnMEifFLRah-lE--CjRxAQ98tZ23/view?usp=share_link

My full log files for 3 days.

[shubhank008]

Mate, since you use Azure, a few things you should try are:

1. Separate your database server to a separate server
2. Separate your map servers to their own individual servers
3. Increase CPU cores to 4 on your main server

Do this temporarily and monitor resource usage on all of these servers, even a few hours or 1 day of billing is fine, that should give you a idea as to which part of the topology is resource hungry and if its mapserver, which specific map is the resource hogger.

[temel61]

I actullay have not a problem with Amazon AWS last week.I moved to azure and having problem.Do you recommend me a server company ?. I will move to my game to new server.I guess I need 32 gb RAM and minimum 4 CPUs.

I have only 1 online and result.

[insthync]

Must find out why it runs so many map-servers like that first, then as I see in the log files, it is trying to spawn monsters every second, when it is going to spawn, it will random position and then raycasting to find the ground position, I've noticed that it is retrying too frequently, try set higher amount of Monster Spawn Area -> Respawn Pending Entities Delay

[temel61]

I will set respawn value as 60 seconds.İf I kill other map-servers everythink works well ?

[insthync]

I don't know.

I told you how to killall before running the servers, if you already did it but it still has so many map servers running like this, then I want to know why it is having so many map servers running first.

[temel61]

There are many database error in my centrol log file.

ERROR No Tag [2023-04-27 07:50:30] - (id!='B8niyTqOQwtR3Zdp') ERROR No Tag [2023-04-27 08:00:41] - (id!='EpsmJz7yRgEynwm') ERROR No Tag [2023-04-27 08:08:09] - (id!='PmtHGIvjakf8bDdu') ERROR No Tag [2023-04-27 08:10:27] - (id!='PmtHGIvjakf8bDdu') ERROR No Tag [2023-04-27 08:11:15] - MySqlConnector.MySqlException (0x80004005): Duplicate entry '8oJYG9lpdj7TQHNv' for key 'characteritem.PRIMARY' at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync (MySqlConnector.Protocol.Serialization.IOBehavior ioBehavior) [0x00066] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\Core\ResultSet.cs:50 at MySqlConnector.MySqlDataReader.ActivateResultSet (System.Threading.CancellationToken cancellationToken) [0x000a7] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlDataReader.cs:135 at (wrapper remoting-invoke-with-check) MySqlConnector.MySqlDataReader.ActivateResultSet(System.Threading.CancellationToken) at MySqlConnector.MySqlDataReader.CreateAsync (MySqlConnector.Core.CommandListPosition commandListPosition, MySqlConnector.Core.ICommandPayloadCreator payloadCreator, System.Collections.Generic.IDictionary2[TKey,TValue] cachedProcedures, MySqlConnector.Core.IMySqlCommand command, System.Data.CommandBehavior behavior, MySqlConnector.Protocol.Serialization.IOBehavior ioBehavior, System.Threading.CancellationToken cancellationToken) [0x00245] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlDataReader.cs:459 at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync (System.Collections.Generic.IReadOnlyList1[T] commands, MySqlConnector.Core.ICommandPayloadCreator payloadCreator, System.Data.CommandBehavior behavior, MySqlConnector.Protocol.Serialization.IOBehavior ioBehavior, System.Threading.CancellationToken cancellationToken) [0x0034d] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\Core\CommandExecutor.cs:60 at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync (MySqlConnector.Protocol.Serialization.IOBehavior ioBehavior, System.Threading.CancellationToken cancellationToken) [0x00064] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlCommand.cs:266 at MySqlConnector.MySqlCommand.ExecuteNonQuery () [0x00000] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlCommand.cs:101 at MultiplayerARPG.MMO.MySQLDatabase.ExecuteNonQuerySync (MySqlConnector.MySqlConnection connection, MySqlConnector.MySqlTransaction transaction, System.String sql, MySqlConnector.MySqlParameter[] args) [0x00057] in <095c7299455b401288d4af81c425c125>:0 ERROR No Tag [2023-04-27 08:11:15] - MySqlConnector.MySqlException (0x80004005): Duplicate entry 'yE8hxaypZJecm8AO' for key 'characteritem.PRIMARY' at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync (MySqlConnector.Protocol.Serialization.IOBehavior ioBehavior) [0x00066] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\Core\ResultSet.cs:50 at MySqlConnector.MySqlDataReader.ActivateResultSet (System.Threading.CancellationToken cancellationToken) [0x000a7] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlDataReader.cs:135 at (wrapper remoting-invoke-with-check) MySqlConnector.MySqlDataReader.ActivateResultSet(System.Threading.CancellationToken) at MySqlConnector.MySqlDataReader.CreateAsync (MySqlConnector.Core.CommandListPosition commandListPosition, MySqlConnector.Core.ICommandPayloadCreator payloadCreator, System.Collections.Generic.IDictionary2[TKey,TValue] cachedProcedures, MySqlConnector.Core.IMySqlCommand command, System.Data.CommandBehavior behavior, MySqlConnector.Protocol.Serialization.IOBehavior ioBehavior, System.Threading.CancellationToken cancellationToken) [0x00245] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlDataReader.cs:459 at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync (System.Collections.Generic.IReadOnlyList1[T] commands, MySqlConnector.Core.ICommandPayloadCreator payloadCreator, System.Data.CommandBehavior behavior, MySqlConnector.Protocol.Serialization.IOBehavior ioBehavior, System.Threading.CancellationToken cancellationToken) [0x0034d] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\Core\CommandExecutor.cs:60 at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync (MySqlConnector.Protocol.Serialization.IOBehavior ioBehavior, System.Threading.CancellationToken cancellationToken) [0x00064] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlCommand.cs:266 at MySqlConnector.MySqlCommand.ExecuteNonQuery () [0x00000] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlCommand.cs:101 at MultiplayerARPG.MMO.MySQLDatabase.ExecuteNonQuerySync (MySqlConnector.MySqlConnection connection, MySqlConnector.MySqlTransaction transaction, System.String sql, MySqlConnector.MySqlParameter[] args) [0x00057] in <095c7299455b401288d4af81c425c125>:0 ERROR No Tag [2023-04-27 08:11:15] - MySqlConnector.MySqlException (0x80004005): Duplicate entry 'n2FZQykl--I4BGZ' for key 'characteritem.PRIMARY' at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync (MySqlConnector.Protocol.Serialization.IOBehavior ioBehavior) [0x00066] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\Core\ResultSet.cs:50 at MySqlConnector.MySqlDataReader.ActivateResultSet (System.Threading.CancellationToken cancellationToken) [0x000a7] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlDataReader.cs:135 at (wrapper remoting-invoke-with-check) MySqlConnector.MySqlDataReader.ActivateResultSet(System.Threading.CancellationToken) at MySqlConnector.MySqlDataReader.CreateAsync (MySqlConnector.Core.CommandListPosition commandListPosition, MySqlConnector.Core.ICommandPayloadCreator payloadCreator, System.Collections.Generic.IDictionary`2[TKey,TValue] cachedProcedures, MySqlConnector.Core.IMySqlCommand command, System.Data.CommandBehavior behavior, MySqlConnector.Protocol.Serialization.IOBehavior ioBehavior, System.Threading.CancellationToken cancellationToken) [0x00245] in D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlDataReader.cs:459

[temel61]

I do not have this path on my pc or server. -> D:\Projects\CSharp\MySqlConnector-1.3.11\src\MySqlConnector\MySqlDataReader.cs:459

[insthync]

For that one you have to update MMO codes, they are all fixed.

Don't find its source code by that path, it's path of MySqlConnector developer's device.

Just do what i have told, UPDATE.

Atleast try this https://github.com/insthync/UnityMultiplayerARPG_MMO/tree/3d228049da62ac172ebf1da58169d9a6823f2692

[temel61]

I decided to update my kit verison to 1.82 from 1.78.I also decided to move azure server to google cloud server 32 gb ram and 8 cpu.I hope it does not cause any lagging.I will let you know the results tomorrow.

[temel61]

I upgraded my kit versiyon to 1.82 and I moved my database to new server in google cloud 32 gb ram , 8 CPUs Nothing changed. This photo without online. Maybe my command is wrong :/

nohup ./Build.x86_64 -startCentralServer -startMapSpawnServer -startDatabaseServer > /dev/null 2>&1&

[temel61]

[insthync]

Yep, that update is fixing database race conditions, not map-spawn issues. As this waste so much time, I think you should allow me to see it, you may enter your server by ssh client then I will use AnyDesk to enter your device