citizenfx / fivem

The source code for the Cfx.re modification frameworks, such as FiveM, RedM and LibertyM, as well as FXServer.
https://cfx.re/
3.57k stars 2.12k forks source link

[Bug] Inconsistent tattoo sync and visibility #2010

Open horejsiladislav opened 1 year ago

horejsiladislav commented 1 year ago

Some tattoos are not syncing to other players, even though they are loaded and streamed correctly. This happens only with some tattoos, and the ones that are not visible depend on the server build. The problem does not seem to be related to exceeding the TxdStore pool, loaded ytds, or the number of addon tattoos ingame. The problem also occurs with base GTA tattoos.

📦Reproduction repo: https://github.com/horejsiladislav/fivem_tattoo_sync_bug 👉 Before submiting any suggestions, please read the details, there was a lot of research and debug before submitting this issue, so some ideas may already be known. Thanks!

âť— Steps to reproduce are in README. Please pay attention to the requirements, each server build has this problem with different tattoos. To see it in action, use server build provided in README to see the bug with tattoos predefined in the reproduction repo. (other build would have same problem only with different tattoos).

image

Details

Also available in details.md in the repo.

Click to expand all details 👇 1. The tattoos you don’t see depend on the server build you have. Best guess for this is that each new dlc adds some tattoos and probably shifts some indexes of loaded textures? 2. Tattoos sometimes aren't visible even on the player themselves, but most of the time, the player can see their own tattoos while others can't. 3. The number of tattoos does not seem to affect the issue. We tried adding different quantities of addon tattoos, and oddly enough, a larger number of tattoos sometimes synced when smaller did not. 4. Initially, we thought the issue could be related to having too many addon textures streamed, but even after loading our server with every addon we could find (including a custom addon clothes pack with over 15000 different ytds), the problem remained. The number of ytds that aren't tattoos doesn't seem to affect the problem. (the ytds were loaded for player, we precisely loaded amount to be a bit lower than TxdStore pool to not crash the game if anything else loads) 5. We tried it on different PCs and various servers (ESX, QB, vRP, clean, with scripts/without, with small/large amount of addon stuff and more..). Same results on all platforms. 6. We made different addon tattoo packs and streamed them as separate scripts for testing. Interestingly, when we loaded e.g. 4 packs and started only 3, the tattoos from the third pack were not visible. When we started the fourth pack (with a server restart to load all textures), tattoos from the third pack started appearing, but some from other packs disappeared. (Another hint to the “index shifting” suggested in first point?) 7. We confirmed the problem is not related to exceeding the TxdStore pool. Adding more ytds (as in point 4) results in a crash due to exceeding the pool size (95500), which does not happen in the case of the tattoo syncing issue. 8. The ytd file for the non-loading tattoo is correctly listed in both players' streaming lists. 9. Manually requesting the ytd for a player who can't see it on another player doesn't fix the issue. The ytd is requested and loads correctly when checked with the native function, but the player still can't see it even after restarting the pack the ytd is from. (Example shown in script provided below) 10. It doesn't seem to be an issue with streaming list _(F8 > Tools > Streaming > Streaming list)_ indexes either. For example, a tattoo with index 35793 wasn’t syncing, but one with index 35818 was. 11. One idea to overcome this was to check for players in close radius, get their tattoos and request all the ytds for yourself. This idea, as you may have guessed, didn’t work as we found out that requesting and loading the ytd is not the problem as said in point 9. 12. If it wasn’t obvious from the points above, we don’t think the addon tattoo files are the problem. They seem to stream just fine when we start different numbers of packs or use them on another server build (where of course some tattoos that worked before stop working). 13. You can actually see the non-syncing tattoo on yourself, so it is loaded for you, but can’t get it to sync from other player to see it on their body. 14. The character with the tattoo has the texture dict loaded (checking with `HasStreamedTextureDictLoaded` ), whereas the second char, that does not see it, doesn’t. (Can be tried out in the testing script with `/tat_request`) 15. One interesting thing we have found while finishing the bug reproduction script, is that the name of the xml files of addon tattoos affect the loading of the textures. (= differently named xml files resulted in different tattoos visible ingame, both base and addon) 16. And the last point where it get’s even crazier.. it does not happen only with addon tattoos, but with base GTA tattoos too. You can see it even in the reproduction package.
gottfriedleibniz commented 1 year ago

There is a limit on the number of presets that can be bRequiredForSync'd (Packed Decorations). Essentially, a bit for each while X*32 bits are reserved in the sync tree. The reason why this works on resource restart is due to an imprecise qsort comparison function when sorting the list of bRequiredForSync'd collections (int overflowing - this is a game-code bug).

It is theoretically possible to increase this limit, but it would probably be best to exhaust all other options before exploring the route (e.g., hijack/replace overlays/textures from previous DLCs).

horejsiladislav commented 1 year ago

@gottfriedleibniz Hey, thanks for the quick response! Sorry for the delay, just got back from vacation. I get that increasing the bRequiredForSync limit might be a big ask. However, considering how fast FiveM development is moving and the trend of using addon tattoos and just addon stuff in general, going the replace route may not be a practical solution for the players.

We've spent a significant amount of time researching the problem and talking to players before submitting this issue and unfortunately, every time we got back to this.

With each new content, it seems like this tattoo sync issue just keeps popping up. So, wouldn't bumping up the bRequiredForSync limit be a more direct, and maybe even a more future-proof solution? Let's be honest, players love their addon stuff – but I am sure you already know that.

Let me know what you think, and a big thank you for looking into this!

horejsiladislav commented 1 year ago

Hey there @gottfriedleibniz, hope you're having a good day! I just wanted to ask you about any follow up on this issue? It is still a problem to this date and with more tattoos coming in newly released builds, it is even worse.

Thank you for your help on this and let me know if I can do anything to assist.

gottfriedleibniz commented 10 months ago

So there are two separate issues here:

  1. PedDecorationManager's inconsistent sorting implementation.
  2. Modifying the sync-bits to increase the number of decorations that can be synchronized.

While I can immediately fix the first issue, the concern is that servers may have already compensated for this exact issue. The second case requires careful handling to avoid an inconsistent state between clients on canary vs. beta vs. release and things of that nature.

This will require some thought (see previous discussion: https://github.com/citizenfx/fivem/issues/838).