search

citizenfx / fivem

The source code for the Cfx.re modification frameworks, such as FiveM, RedM and LibertyM, as well as FXServer.
https://cfx.re/
3.53k stars 2.08k forks source link

Triggering latent events on everyone causes network thread hitches #1360

Open manzarek123 opened 2 years ago

manzarek123 commented 2 years ago

When using TriggerLatentClientEvent to a lot of players (tested with ~1000 players) the network thread will freeze and everyone will timeout and be disconnected.

The same event to everyone with TriggerClientEvent works fine. The same event with TriggerLatentClientEvent to a single player works fine as well.

Event data was 47047 Bytes and bps was set to 1000000: TriggerLatentClientEvent('eventname', -1, 1000000, data)

I have and ETL trace, not sure if useful for this issue Trace might have unusual activity as a lot a players where reconnecting when I provoked the crash for the ETL record.

https://1drv.ms/u/s!Asx7IOhJxcoWhZZSCwUrWR_YJTBcAw?e=OcjxPz

[ citizen-server-impl] network thread hitch warning: timer interval of 370 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 445 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 443 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 435 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 435 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 440 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 438 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 451 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 456 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 456 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 442 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 454 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 438 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 452 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 449 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 449 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 425 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 462 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 446 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 436 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 435 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 452 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 467 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 477 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 439 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 440 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 458 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 454 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 464 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 432 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 453 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 442 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 441 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 437 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 456 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 432 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 420 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 439 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 441 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 451 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 448 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 445 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 435 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 449 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 465 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 477 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 445 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 442 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 428 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 451 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 458 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 446 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 463 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 464 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 457 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 467 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 441 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 463 milliseconds
[ citizen-server-impl] network thread hitch warning: timer interval of 438 milliseconds
blattersturm commented 2 years ago

At this size/bps it's likely triggering a lot of immediate resends (even worse if any transmission actually dropped fragments).

Sadly, this isn't a very reproducible scenario, so this would need someone who does have the ability to work on and test network stuff like this to fix it.

GlitchOo commented 2 years ago

I can confirm this was a problem with just a 100 players. TriggerLatentClientEvent was replaced with just TriggerClientEvent and the issue has stopped.

blattersturm commented 2 years ago

Instead of 'confirming' and not providing any additional info to make this actionable, how about providing the info asked for, or testing with less nonsense bps values, or whatever? 😕

GlitchOo commented 2 years ago

Instead of giving a snarky sarcastic reply that does nothing but encourage the exact opposite how about you actually ask for more information in your previous post instead. I don't see any. If I had more information to give then I would... I was simply confirming this isn't an issue with just one person so maybe it could be investigated more. BTW it was tested with just 250, it made no difference.

blattersturm commented 2 years ago

Sigh. Another issue lost to the trolls that start demanding and arguing in weird non-constructive ways.

If this issue were able to be investigated, it already would be, and probably would already be fixed. 'Confirming' to show 'please investigate!' doesn't add anything of note, and it only makes for more annoyance since there isn't any means for us to investigate this.

We don't have access to any server with even 5 active players, let alone your 'just' 100 players, nor do we have access to your code, or any of the other setup involved in replicating this. That is also what the initial response was pointing at: this requires such specific context to be able to 'investigate' and fix, that such a fix can only be done by an affected server owner who can test stuff themselves.

Again, 'same issue' style responses do not help. An issue does not magically become more actionable because more people are reporting it. We don't discriminate issues based on 'how many reports there are', and rather, the more reports there are for a non-actionable issue without reproduction steps, the more likely it is to be ignored as it becomes a sensitive subject to even think of touching.