search

citizenfx / fivem

The source code for the Cfx.re modification frameworks, such as FiveM, RedM and LibertyM, as well as FXServer.
https://cfx.re/
3.41k stars 2.01k forks source link

Sync + Server thread hitch starting with artifact 8096 #2557

Closed manzarek123 closed 2 weeks ago

manzarek123 commented 1 month ago

What happened?

Starting with artifact 8096, every time a new player is connecting to the server there is a sync + server thread hitch. This happens at high player count and every time a new player is connecting to the server. People are complaining of more lags but is is probably because there are lots of new players connecting all the time.

This started with build 8096, I have tested the following builds: 7900 ok 8075 ok 8085 ok 8096 not ok 8139 not ok 8181 not ok

Let me know if I can help to provide more info

[ script:bay_queue] [Queue] Allowed jeuke from VIP Queue (size: 2) [ citizen-server-impl] sync thread hitch warning: timer interval of 185 milliseconds [ citizen-server-impl] server thread hitch warning: timer interval of 195 milliseconds

script:bay_queue] [Queue] Allowed Satoshi (Fr) from VIP Queue (size: 1) [ citizen-server-impl] server thread hitch warning: timer interval of 171 milliseconds [ citizen-server-impl] sync thread hitch warning: timer interval of 109 milliseconds

[ script:bay_queue] [Queue] Allowed ogu from Queue (size: 43) [ citizen-server-impl] server thread hitch warning: timer interval of 195 milliseconds [ citizen-server-impl] sync thread hitch warning: timer interval of 215 milliseconds

Expected result

No sync and server thread hitch

Reproduction steps

  1. Have a server wiith arctifact 8096
  2. Connect to the server

Importancy

Slight inconvenience

Area(s)

FXServer

Specific version(s)

Server 8096

Additional information

No response

gottfriedleibniz commented 1 month ago

Would it be possible to get a ETW Trace? May be possible that reverting Lua back to use the systems allocator is causing a bit of lock contention.

manzarek123 commented 1 month ago

I will update to 8096 this afternoon and make the trace this evening !

manzarek123 commented 1 month ago

Here is a trace, let me know if it is good https://1drv.ms/u/c/16cac549e8207bcc/Ea7_wWYPgS1JqcQVSYq_5w0B5InYZvxSJ6cJ-jb0Q48nKQ?e=dQD7KC

gottfriedleibniz commented 1 month ago

Thanks. Seems to be the case: garbage collection step deallocating ~800 objects and this is contending with some other threads. The Windows allocator is pretty slow in general so Lua's custom allocator was probably carrying a lot of weight.

This trace showed a few areas where heap interactions can be reduced/optimized. Hopefully I'll be able to get to this sooner rather than later.

Korioz commented 1 month ago

Up, this affects a lot of servers, we cannot update to newer builds because of those massive hitches, this cause our players to see this misleading "Server artifact version is outdated" message.

image

gottfriedleibniz commented 1 month ago

Would it be possible to test artifact 8450. If you are still running into issues, would it be possible to send another ETW trace?

There's still some heap contention between the other threads that needs to be optimized.

manzarek123 commented 1 month ago

Sure, I'll update now and report this evening when player count is high

manzarek123 commented 1 month ago

there are no issues with build 8450