Open MaaximGL opened 2 months ago
Wanted to add, eventually most/if not all the textures will do this all over the map and not just marked location, at least from what I found. This is either due to being on a server too long(like 4-5 hours and it will start) or if its more populated, it happens fairly quickly.
Wanted to add, eventually most/if not all the textures will do this all over the map and not just marked location, at least from what I found. This is either due to being on a server too long(like 4-5 hours and it will start) or if its more populated, it happens fairly quickly.
We have definitely had this happen within just a few minutes, even with just one player, if you fly about as MaaxinGL explained it. If not actively trying to get it to bug out, it sure will take a bit longer.
Applying these settings, as MaaximGL also mentioned from my post on the forum seem to have fixed it for me, however not for one of the others on my server - so no exact science I suppose:
Hello! 😄 We're experiencing the same issue on SOZ, with 240 ~ 300 players connected simultaneously daily.
After our recent update, the majority of players encountered this problem. Some could play for several hours without any issues, while for others, the problem appeared within 2 minutes of playing.
We've tried everything, including reducing the mapping load and optimizing asset appearances, but unfortunately, we haven't been able to solve the issue. :/ Since we couldn't find a solution, we've reverted to an earlier update.
We also use GTA V Remastered & Forests South + North, and from what I've read, that could be part of the problem too.
We also use GTA V Remastered & Forests South + North, and from what I've read, that could be part of the problem too.
We also use a similar map mod: https://github.com/IDKFORCE/fs-serversidegraphics, although we never had this issue before this build.
But the issue seems to be almost random. Sometimes it's happens almost instantly, sometimes never at all. We've been trying to pin down a reproducable method for how to cause this to happen, but it's very random. Some of our users experience it all the time, some never. 10-20 players on average.
I will add that what used to work for me was the aforementioned Extended Distance Scaling (0) and Distance Scaling either or. But now, especially as of last night - every other building and area is doing it. I would move my ped all of 2 buildings away and suddenly its messed up. I was only on server for 5 minutes. Computer was freshly restarted. Server was fresh restart as well. The amount of players didn't matter. Its been getting progressively worse over time.
As it seems to happen on a clean server as mentioned above I don't think its specific to add ons at this moment. Though, those certainly can contribute to the issue! Especially with how lazy some mappers are, and how insane their LOD distances are, or broken LOD indexes!
This issue however is literally everywhere for me and others not just in a few key spots.
Continues to be a significant issue for much of our playerbase, making b3258 unplayable for some users without having to restart several times.
Is there any idea on a fix for this or should we revert?
Yeah we are getting SO many complaints. Sometimes immediately when loading in I even get the issues myself. Never used to happen THIS badly. What did Rockstar change with SLOD2 models or loading or index values on the last update? Some way to easily see?
ps. @TheIndra55 does a lot of work on bob74 and maybe has some advice. Sorry for the mention but I respect all the work you do on that Repo and perhaps you may have some knowledge and insight.
We ended up downgrading back to the previous gamebuild. It's definitely just the newest gamebuild that causes texture loss/playdough. Hopefully it's fixed sooner than later.
How to fix this? Downgrade game build and artifact? Or just the game build?
How to fix this? Downgrade game build and artifact? Or just the game build?
The fix would be to downgrade yes. The other fix would either be from CFX or if someone knows how, some kind of community solution.
I assume the core issue is that map makers are not properly making their maps, but something before was clearing/stopping this from being a widespread issue.
How to fix this? Downgrade game build and artifact? Or just the game build?
The fix would be to downgrade yes. The other fix would either be from CFX or if someone knows how, some kind of community solution.
I assume the core issue is that map makers are not properly making their maps, but something before was clearing/stopping this from being a widespread issue.
Well some people have reported this issue happening on a clean server with no custom ymap/mlo's
Not sure if this is related, but the ability to increase pools seems to be coming in the future:
https://github.com/citizenfx/fivem-docs/pull/481/files
However, this may not solve the issue being experienced here.
Please Fivem teams Fix this ❤️
Not sure if this is related, but the ability to increase pools seems to be coming in the future:
https://github.com/citizenfx/fivem-docs/pull/481/files
However, this may not solve the issue being experienced here.
We are running all of those pool size values increased with several of our players on Canary build to test and unfortunately the problem remains.
One REPRODUCTION spot which is a specific spot that is a constant source of the issue is near Braddock Pass tunnel.
Our server is not streaming this file but it still happens VERY frequently where it will not unload.
cs1_lod3_terrain_slod2_17
with 14 children. Has 761741358 GUID.cs1_lod3.ytyp
hei_cs1_lod3.ymap
Perhaps there is a conflicting LOD Index somewhere but with no feasible way of checking every LOD Index to see if there is a conflict then I'm not sure.
We know that Extended Distance Scaling lower and lower Distance Scaling helps the issue a little bit. So then I must ask what if its related to the calculation of the PlayerPed current coords in the world kind of messing up and not being accurate therefore the LOD's don't load or unload properly.
Here are my Pool Sizes right now, just loaded in not even 2 minutes on the server. Near the Wind Farm of all places where we don't have any mods whatsoever.
This still remains a major issue for anyone using this build, I hope someone in CFX can bless us with a fix soon 🙏
This still remains a major issue for anyone using this build, I hope someone in CFX can bless us with a fix soon 🙏
Indeed to the point where people are leaving servers over it thinking that its just specific to that server. I've seen a few owners talking about it in other places as well who are unaware of this repo or the forums.
This still remains a major issue for anyone using this build, I hope someone in CFX can bless us with a fix soon 🙏
Indeed to the point where people are leaving servers over it thinking that its just specific to that server. I've seen a few owners talking about it in other places as well who are unaware of this repo or the forums.
There seems to be a new build 3323. Has anyone checked if this fixes the issue?
There seems to be a new build 3323. Has anyone checked if this fixes the issue?
Can confirm after testing for around 1 hour on a fresh install and also 1 hour on our test environment that the issue is still occurring on 3323.
Someone posted this in one of the forum posts about this issue. Apparently someone for single player decided to make a fix (though this one is currently outdated and not made for FiveM)
• Source: https://github.com/0x-FADED/GTA-V-fwBoxStreamerVariable-and-decals-limit-Patch • Compiled: https://www.gta5-mods.com/tools/ymap-load-list-extent-limit-fix-fwboxstreamervariable-patch
Perhaps this can help if adapted for FiveM and updated?
Someone posted this in one of the forum posts about this issue. Apparently someone for single player decided to make a fix (though this one is currently outdated and not made for FiveM)
• Source: https://github.com/0x-FADED/GTA-V-fwBoxStreamerVariable-and-decals-limit-Patch • Compiled: https://www.gta5-mods.com/tools/ymap-load-list-extent-limit-fix-fwboxstreamervariable-patch
Perhaps this can help if adapted for FiveM and updated?
I believe this patch is already part of FiveM. If you go to the dllmain part of the source code (https://github.com/0x-FADED/GTA-V-fwBoxStreamerVariable-and-decals-limit-Patch/blob/main/dllmain.cpp) you can find a comment: made originally by CitizenFX and this is part of FiveM I have just ported it into standalone asi/dll
. There are also links to the FiveM sources that look identical.
Tho it still may be related. Maybe increasing those limits even further will help.
We're also seeing this on our server with 3258, when opening the streaming memory with strmem 1
im seeing ytd's staying loaded for significant portions of the map. Even in paleto im seeing assets for MBA and MRPD map still loaded. I'm guessing thats probably related to the slods not unloading but figured id mention it.
We're also seeing this on our server with 3258, when opening the streaming memory with
strmem 1
im seeing ytd's staying loaded for significant portions of the map. Even in paleto im seeing assets for MBA and MRPD map still loaded. I'm guessing thats probably related to the slods not unloading but figured id mention it.
Allow me to clarify its not an issue with textures as in (.ytd) files. Its an issue with models (.ydr). Even though people refer to the issue commonly as "texture loss" or "blurry textures" it is in fact blurry textures loaded on a low poly model that doesn't unload properly from view.
SLOD's are models with blurry low quality textures and low poly counts to be shown at a distance such as when in a plane high above or far away where detail doesn't matter but optimization does.
However, molo's work is often unoptimized and I can attest to that having bought several of his maps. Its likely that its got really far set LOD distances in the .ymap's and .ytyp files from his MRPD thus causing textures to load for them at great distances. This is a common problem with many mappers who don't understand optimization, nor care about it or even know what LOD's are.
A hot tip on the side of .ytd themselves, if those are really high quality such as 2k and 4k textures inside of it and the .ytd is massive as a result, that will cause a different issue at times (usually when combined with tons of others that are oversized) which is people running out of VRAM/RAM where you will see entire models be invisible. But thats a different issue than this issue we are talking about. Accompanying those errors you will see "asset is oversized" warnings in yellow and red in the server console.
Update: this problem immediately appears on b3258. Also met on b3095, but much less often. It was not seen on versions before b3***.
Just wanted to give a quick update about this issue. I've been working on it for some time. I can't tell when it will be done - it depends on how deep the root cause is. But fingers crossed soon.
It does reproduces on completely clean server. But additional map resources make it appear faster. For the same reason it appears faster on the later game builds.
The blurred LODs are not unloaded because some of it's children are failed to be created. Usually its something small, like a few bushes - so it creates an illusion that LOD is displayed for no reason. And the children are not created because of a memory leak in fwArchetypePooledMap
(its not tracked by Pool monitor
, so you don't see when it happens). Why the leak is happening - thats a question that I'm currently investigating. My current assumption - a race condition, but I couldn't confirm it yet.
I also can imagine a map so packed with additional resources that the archetype pool will get overloaded right away. Unfortunately - increasing it's size is tricky. Because the base game uses int16 to store the size it and current size (64000) is already very close to the data type size limit (2^16). Its also referenced in a lot of places. Increasing it will be a task for the future for sure, but fixing the leak is a higher priority for me right now.
Thank you for your focus on this @Nobelium-cfx 🙏
int16 to store the size it and current size (64000) is already very close to the data type size limit (2^16)
Thank you for looking into this! @Nobelium-cfx
Off Topic: Ah yes, int16 to store the size it and current size (64000) is already very close to the data type size limit (2^16)
- the bane of my existence I feel like. Thats including the other semi-related, dreaded, carcols siren value id limit of 255. Due to it being only 8-bit as well. It too needs an increase some day to at int16 we hope. There was another instance of int8 > int16 where an update to this value was made for EUP issues which was nice. That one resolved the invisible hat props on servers with more than 255 of them.
Not sure how useful this is for debugging purposes, but there is an older PR that added an archetype viewer to devtools: https://github.com/citizenfx/fivem/pull/2240 Sadly, this never got merged, but if you're able to build a custom version of FiveM, I rebased this on the current HEAD: https://github.com/tens0rfl0w/fivem/tree/feat/devtools/archetype-viewer
Hi, thank you! It does look useful. I've collected very similar data when debugging. But had to implement it myself. Will for sure keep this tool in mind. But I believe I've already found an issue. Or, at least the main one of them. Which makes it much more common on build 3258+.
I was going to write an update when the fix is merged. But it won't happen till Monday and I've figured that people may spend some time on it during the weekends. So it's probably best to give the current status.
There is a memory corruption. It happens here: https://github.com/citizenfx/fivem/blob/master/code/components/gta-streaming-five/src/PatchStreamingPreparation.cpp#L538 . _fwEntity_SetModelId for build 3258+ accesses data at ((uintptr_t)fakeEntity + 44), which is outside the memory that we allocate.
After the fix I could no-clip around the island for hours without any visible issues or sign of a leak (I added some debug code to detect it). On clean build and with some additional resources (mostly https://www.gta5-mods.com/maps/gta-v-remastered-enhanced and https://www.gta5-mods.com/maps/forests-of-san-andreas-revised). Before the fix such stress test would lead to the problem within 10-15 minutes (up to half an hour).
So I was wrong about my statement that on 3258+ the issue appears to be more often simply because there are more content and therefor more archetypes. I know that people pointed that they have seen the same issue on earlier builds. Tbh I just assumed that it probably the same issue. I've briefly tested 3095 today - and I couldn't reproduce the issue. Tho I think I did see it when I just started working on this and was establishing some good way to reproduce the issue. So its probably there, but much less prominent.
I'm pretty sure that the base mechanism for this issue on earlier builds is the same - some children of a lod are not loaded, so the parent lod stays visible. But why it happens - Im not sure. I have two theories:
The fix should now be available in all branches (canary, beta, prod). Let me know if you still see this issue.
Will keep the bug open for now.
This is amazing work, @Nobelium-cfx!! <3 Thank you so much for your effort on this one! This will help so many people, even on lower gamebuilds! <3 Will give feedback if we spot this one again! 🙏
The fix should now be available in all branches (canary, beta, prod). Let me know if you still see this issue.
Will keep the bug open for now.
A gazillion thanks for the work! We will be testing it tonight. Some nuts are tough to crack, let's hope we got this one! ;) Running b3258 with Windows artifact 10309 :)
Amazing work @Nobelium-cfx!
We were debugging this for a while thinking it was a local resource causing the issue. Glad this was solved in the end.
Indeed
The fix should now be available in all branches (canary, beta, prod). Let me know if you still see this issue.
Will keep the bug open for now.
@Nobelium-cfx
After the update people are getting Timeout detected on clientside lua interface.
Do you think it could be related to this commit or something else? Started happening with some clients only after updating to 10309 (3258)
The fix should now be available in all branches (canary, beta, prod). Let me know if you still see this issue. Will keep the bug open for now.
@Nobelium-cfx
After the update people are getting
Timeout detected on clientside lua interface.
Do you think it could be related to this commit or something else? Started happening with some clients only after updating to 10309 (3258)
There are some potential issues in Discord reported with the latest build, it might just be the artifact.
Switching to Canary seemed to fix it. But everyone on release, including me, would load in and then get stuck in loading screen. I see this.
[ 1217969] [b3258_GTAProce] MainThrd/ spawned
[ 1217985] [b3258_GTAProce] MainThrd/ Error invoking callback for event playerSpawned: System.InvalidCastException: Invalid cast from 'System.Double' to 'System.Dynamic.ExpandoObject'.
[ 1217985] [b3258_GTAProce] MainThrd/ at System.Convert.DefaultToType (System.IConvertible value, System.Type targetType, System.IFormatProvider provider) [0x00269] in <b575c7ac070e41ffa0751333b6ac9f42>:0
[ 1217985] [b3258_GTAProce] MainThrd/ at System.Double.System.IConvertible.ToType (System.Type type, System.IFormatProvider provider) [0x00000] in <b575c7ac070e41ffa0751333b6ac9f42>:0
[ 1217985] [b3258_GTAProce] MainThrd/ at System.Convert.ChangeType (System.Object value, System.Type conversionType, System.IFormatProvider provider) [0x0021b] in <b575c7ac070e41ffa0751333b6ac9f42>:0
[ 1217985] [b3258_GTAProce] MainThrd/ at System.Convert.ChangeType (System.Object value, System.Type conversionType) [0x0000c] in <b575c7ac070e41ffa0751333b6ac9f42>:0
[ 1217985] [b3258_GTAProce] MainThrd/ at CitizenFX.Core.CallUtilities.<GetPassArguments>g__ChangeType|0_0 (System.Object value, System.Type type) [0x0002d] in C:\gl\builds\cfx-fivem-1\code\client\clrcore\EventHandlerDictionary.cs:149
[ 1217985] [b3258_GTAProce] MainThrd/ at CitizenFX.Core.CallUtilities.GetPassArguments (System.Reflection.MethodInfo method, System.Object[] args, System.String sourceString) [0x000d0] in C:\gl\builds\cfx-fivem-1\code\client\clrcore\EventHandlerDictionary.cs:209
[ 1217985] [b3258_GTAProce] MainThrd/ at CitizenFX.Core.EventHandlerEntry+<Invoke>d__5.MoveNext () [0x00047] in C:\gl\builds\cfx-fivem-1\code\client\clrcore\EventHandlerDictionary.cs:106
Switching to Canary seemed to fix it. But everyone on release, including me, would load in and then get stuck in loading screen. I see this.
This definitely doesn't look related to my changes. But maybe it was caused by something that was pushed alongside with them. I'll double check.
@Legacy-TacticalGamingInteractive I can't see anything that could cause such error. If I'm not missing anything - playerSpawned
is only triggered in resources/[managers]/spawnmanager/spawnmanager.lua and used in custom resources and scripts. So I can't see how the fivem update could cause any issues with it. Also - if it was production push related - I would expect much more people to be affected.
Are you sure you didn't make any changes in your resource folder that could cause it? Maybe some invalid spawn points were added? Or some new playerSpawned
event handlers?
If the issue persists - please create a new github issue and we can take it from there. So we don't dilute this one with unrelated info.
Is anyone still experiencing this issue? I've had a couple of users report this, but it could be their settings/configurations. I was personally unable to recreate it.
I suspect this is fixed and it's our fault/user error, just seeing if anyone reports the same.
Edit: For anyone coming across this, it was user error.
It happens way less frequently for sure. Twice for me yesterday but I need to test more myself and also tweak settings to see. Requires more time and testing though ultimately. I am waiting for more feedback as well. But its certainly reduced it would seem.
@Legacy-TacticalGamingInteractive I can't see anything that could cause such error. If I'm not missing anything -
playerSpawned
is only triggered in resources/[managers]/spawnmanager/spawnmanager.lua and used in custom resources and scripts. So I can't see how the fivem update could cause any issues with it. Also - if it was production push related - I would expect much more people to be affected.Are you sure you didn't make any changes in your resource folder that could cause it? Maybe some invalid spawn points were added? Or some new
playerSpawned
event handlers?If the issue persists - please create a new github issue and we can take it from there. So we don't dilute this one with unrelated info.
Replying here @Nobelium-cfx just for documentations sake - though unrelated to this issue perhaps. I could open a new issue so we can track it better?
This turns out to be caused by something in the artifacts absolutely causing net game errors, I don't know what, which then caused our anticheat to think someone was exploiting as they were loading in typically, and thus would kick them. We've disabled the kick action from the anticheat to compensate for now as a temporary workaround.
However the server failing to receive net game events is ongoing - we don't know what its source is because we have nothing on server side or client logs about it. It only started happening to about 50% of our players directly with 10309 in regards to the kick issue from Lua interface timeout.
*We did not change any settings or anything on our end we only cleared cache and updated artifacts.
A bit further back however, we do still have a rare issue with Ran out of rage::netGameEvent pool space.
client crashes with some people, even though we've increased it to 911 using the new convars on our end - not everyone is running Canary so it would seem they still reach the limit. We are trying to figure out why thats happening as well such as if its some script maybe. Because its recent too. Could be related directly. Unfortunately though I don't know what artifacts it was that it started to happen more frequently on. May not be related to the above?
I could open a new issue so we can track it better?
Yes please.
I don't have any ideas at the moment. I have little expertise with the network layer unfortunately. Maybe @FabianTerhorst knows something.
But lets continue the discussion in the new bug.
Using an author's paid map extension, 'Roxwood,' seems to make the texture loss more frequent. I'm not sure if I'm allowed to link it, but it's a popular one. I'm also on build 3095.
Using an author's paid map extension, 'Roxwood,' seems to make the texture loss more frequent. I'm not sure if I'm allowed to link it, but it's a popular one. I'm also on build 3095.
Hi @Julian12it
I will note that Roxwood is a completely unoptimized. I'm actually shocked at how bad the optimization is. I wouldn't read too much into that. While it can/will cause crashing and texture issues thats because the mod itself is not well made.
Theres a reason why they decided to make the next patch for it focus on optimization.
Some of those include 50,000 meter LOD's on things like beer bottles and tiny props to everything else. As well as default GTA props reuploaded, renamed and added to their .ytyp for no reason which further adds to the total streamed assets count on the server. There are also major bugs with scenario files, path nodes, .gfx files and more in there. There are also issues with physics validation on poly edge references on many props. I have had to heavily customize Roxwood just to integrate it properly on our end in our use case. I am looking forward to the next patch they release and I have brought forth the issues with them in tickets.
Still receiving similar results even after removing Roxwood. See blow:
Still receiving similar results even after removing Roxwood.
Then you are using other maps that are not optimized / badly made. Got + 50 MLO with ~20 vehicles + very high cloth pack and no problem at all
Still receiving similar results even after removing Roxwood. See blow:
Can you give me a list of resources that you are using? If you narrow it down to some small amount - that would be very helpful for investigation :).
Also, how fast after you start the game and how often does the issue appear for you? Any specific actions that seem to trigger it?
I've been running additional stress tests on a clean server recently and couldn't reproduce the issue anymore. So it's probably some specific type of resources that cause it (could still be our fault tho, i.e. not handling some cases correctly). Or maybe some specific actions that needs to be done.
I've fixed another issue that could case this bug. See https://github.com/citizenfx/fivem/commit/0e05dbbc0fe962b45b1748911988d8d27b8735c6 .
As I suspected earlier (https://github.com/citizenfx/fivem/issues/2752#issuecomment-2407826030), we did not handle correctly duplicate archetypes in different fwMapTypes assets. Which would case another leak in the archetype pool. If server uses resources with duplicated archetypes.
Note: I'm not 100% sure if duplicating archetypes is a normal thing to do. After this fix it should work. But I would discourage this practice as the base game clearly did not intend this use case.
The fix should be available on Canary soon. Please try it out and let me know if you still see this issue.
I've fixed another issue that could case this bug. See 0e05dbb .
Unfortunately it caused crash on canary and had to be reverted. Investigating the problem. Will let you know when it relands
Unfortunately it caused crash on canary and had to be reverted. Investigating the problem. Will let you know when it relands
Maybe the best is to stop the resource that try to dup existing archetype and send a message in console. (if possible)
What happened?
When switching to b3258, the SLOD2 models (not LOD or SLOD1) started sticking. None of the resources related to b3095 were changed. The problem is not affected by whether the model resources on the server are enabled or disabled (this happens even on a completely clean server). A more open discussion on this topic can be found here: https://forum.cfx.re/t/lod-slod-texture-issues/5074672/116
Expected result
Rendering only HD models, without SLOD2 overlay when standing on the model.
Reproduction steps
Importancy
Crash
Area(s)
FiveM
Specific version(s)
FiveM 9585
Additional information
The issue occurs less frequently or not at all with the following game settings: Settings → Graphics → Distance Scaling (Somewhere between 0 and 75) Settings → Advanced Graphics → Extended Distance Scaling (0)