Open freak2fast4u opened 8 months ago
Also, it seems to almost always happen during a battle, meaning it could be related to some specific graphical effect.
The .dmp file I provided in the zip file was taken using Process Explorer, and I could do this because the game process hadn't actually crashed, but was left running in a wierd state. The game itself doesn't produce dumps when it crashes, so we're lucky here :)
Hi @freak2fast4u can you share me your save file?
Sure thing, game slot 09 (located in slot08 folder because 0-indexed naming scheme) of following zip is the best spot to start grinding the fights until it happens : https://drive.google.com/file/d/1neQLjp_igUpr6TAVvyriETAyQaZ923Fz/view?usp=drive_link
Unzip inside Documents\My Games\FINAL FANTASY XV\Steam\<some_preexisting_id>\
, or cherrypick whatever you know you need.
Move along the edge of the mountain to encounter some wild ennemies and the empire's army units (these are tougher), see map here : https://drive.google.com/file/d/1H_5IWRXaq7Knni40csybeba5wBkw19lB/view?usp=drive_link
I tried reproducing it in that same spot this evening and it took a grand total of 90 minutes of non-stop grinding before it freezed for good. During those 90 minutes of gameplay, there were some 1 second pauses that correspond to event ID 4101 in the event viewer (igfxnd stopped responding), and some graphical glitches in the sky. There is no particular entry in the event viewer correlated to the crash AFAICT.
Here's what I captured this evening (except the videos, these are from yesterday in that same spot, because for some reason I forgot to press record this evening -_-) :
What I find bizarre is the amount of RAM used at the moment of the crash :
Here is what it looks like when the save from slot 09 is freshly launched (and a few moments of loading assets later) :
Latest 5081 WHQL driver seems to behave a bit better, and GPU memory seems to be reclaimed more aggressively according to the in game perf counters.
Also this looks sane even after almost 3 hours of game process run time, the working set didn't balloon to 16GB like previously :
I still get an absurdly high amount of "igfxnd had stopped responding" 4101 events and 1 second freezes mid-battle, but no crash so far. I'll keep going at it ;)
My ... first crash after over 5h straight of non-stop gameplay, using the latest driver (5081 WHQL according to Arc Control) :
The situation definitely improved, and the working set isn't ballooning like crazy but it does increase a fair bit (3.5GB after a fresh game load, and around 5-7GB during gameplay depending in the area, as opposed to 17GB with the previous driver after only 90 minutes.
I found a setting here, provided by the game engine :
... which I believe caps the GPU VRAM amount that can be allocated for textures, and this is the "Very high" value from the game preset, so it's relatively low for a "high" requirement IMHO, but the game came out many years ago, so I don't blame Square for being conservative about this.
I'll try raising that limit and see if I can play more hours without a crash. Would it be possible that the intel arc driver (or DirectX itself) isn't happy swapping massive textures from VRAM to RAM and back ? I could understand low performance, but not a crash, if that's the case. That would mean some component isn't resilient to VRAM pressure somewhere.
I can also share the new savegame and the full memory dump for this crash if required, but I'll have to move stuff around on my google drive first. Maybe comparing yesterday's dump (previous driver) and this dump (current driver) would give more clues, in case you haven't found anything yet.
Edit : lowering the value to the minimum (0, as per the low preset, all other settings being very high) generated a crash 45 minutes into a game (again, during a battle scene, it's always in battles, never free-roaming the map) :
, all relevant files for this specific "low vram scenario" crash here : https://drive.proton.me/urls/0TK44NG6D0#JznUR5KDu2b0
Use game slot 04 (savegame is in slot3 folder), go down the stairs and follow the path to the monsters, game crashed 10s into battle, after 45 minutes of total gameplay.
I'm overriding the TextureStreamingMemory value to 12000, let's see what that does :p
I managed to get it to crash again within 9 minutes of gameplay in a specific area. I'm using all highest settings @ 1440p except for TRAM set to Low (which means TextureStreamingMemory=0), and all NVIDIA tech disabled.
Here are some graphs I could reconstruct from the raw CSV data collected by Intel System Analyser (I can't get it to do a proper full capture for more than a few seconds, and even then, the Graphics Analyzer can't open it, CSV capture is the only thing that works).
Raw CSV : IntelSystemAnalyzer_csv_raw_capture_9_minutes.zip
New savegame : https://drive.proton.me/urls/BT4XBW9ZW8#hWRvlECpBmYc
Use game bloc 04 (from folder slot3) game bloc 07 (from folder slot6), walk north and south from where the party is currently standing, and within less than 10 battles it should crash.
@Arturo-Intel : is there anything else I can do to help ?
Edit : oh, I forgot to mention I've been using the 4K texture pack for the last couple of days (it's a free DLC), which didn't solve anything obviously :p
Also, here's what I meant by 1 second driver pause : https://youtu.be/8VpjDha1FhE (no 3D, hud displayed, main game loop waiting for the gfx driver to reset before resuming battle)
It always comes with an event id 4101 "igfxnd stopped responding". Contrary to what I thought at the beginning, a crash doesn't produce this, but there would have typically been some occurences of these pauses just before an actual crash.
@freak2fast4u thx for all the evidence you provide, with the last one I think we can create a solid case. Need to check on the lab first. Thx for sharing the save file
-- r2
I haven't tried FFXV on Arc yet, but I do know when TRAM is set to highest, it will burn through memory like nothing else. I experienced that badly when I was playing it with my Vega64 at the time. Even if I set HBCC to make the Vega64 behave like it had 16GB of memory, it would go through these phases of:
I did run the Special K fix, and it helped a bit, but lowering the TRAM down to high helped more. I remember back then trying DXVK with it, and it tamped down on the VRAM usage to some degree. It didn't yo-yo as much, but there were still hitches. I may try and revisit it again because at the time, there wasn't an async or gplasync fork back then.
Luckily enough on Arc the VRAM gets reclaimed frequently so this isn't an issue here. Your experience is proof AMD's drivers weren't top notch at the time either :p
Running the game with TextureStreamingMemory=12000 (high is 4000) definitely smoothens the experience for me, and total VRAM usage reported by the game rarely goes over 10000 even in graphically dense areas, but I haven't had the occasion to go through a long enough session to see if it prevents crashing all together. Also this isn't a workaround anybody with a 8GB VRAM GPU can use. Running the game with DVXK definitely prevents crashing but even that stutters quite a lot (I tried both gplasync and vanilla variants, they feel different but both stutter).
There's one thing I forgot to mention, the game is installed on a mechanical HDD for now, though any sane game engine would handle that gracefuly, and funnily enough, there's very little texture pop-in, even with low TRAM values and using a HDD, so the engine itself seems to be handling textures good enough. As I move around the world, I can see the HDD throughput meter go up and down, demonstrating the game is loading stuff dynamically.
During battles, is where the Vertex Shaders get used and abused for all the special effects going on during a battle (and there's a ton !). I've had battles with 20 moving enemies on screen at a given time (+ the 4 party members) and that is usually where it stutters/freezes up the most (which is typically when you don't want it to :p). Yeah, past a certain point in the game, the empire units just won't let up lmao.
Thanks for suggesting Special K, will look into it ;)
Oh well, using DXVK 2.3 (vanilla, not gplasync) also led to the same type of crash, during a battle too. Preceding that crash was what I think is called "vertex explosions" (geometrically distorted graphics, textures popping out of their usual boundaries to the point where geometry isn't recognisable, etc.). DXVK logs were as follows after the crash : ffxv_s_d3d11.log
I will report this to the DVXK devs separately, there might be a quirk on their side too.
@freak2fast4u -- can you give DXVK one. more. try. but using this version just to see if you get better results. For some strange reason, we sometimes fall back on this particular fork and it often comes with some good results.
@freak2fast4u -- can you give DXVK one. more. try. but using this version just to see if you get better results. For some strange reason, we sometimes fall back on this particular fork and it often comes with some good results.
Excellent suggestion ! I still am trying out stuff, and it turns out I tried 1.10.3-async (and vanilla) a few days back and they both work, though it still stutters more than I'd like (async stutters much less than vanilla of course, but still a fair bit). I got past a long and tricky dungeon in one go thanks to that :D
I'm currently using dxvk 2.1 vanilla and it seems stable so far. Still a few stutters here and there but no crash that I can think of. I might give 2.1-gplasync a try next time I play.
I'm yet to open a ticket with the dxvk devs about 2.3 crashing FFXV, I'll scour around and see what useful info I can gather from a crash to give them, the dxvk default logs aren't too helpful (they report the vulkan device has dropped basically ...).
Latest intel arc drivers (5186 published yestderday) still crash after a while, but they're buttery smooth otherwise.
@LancerDawg75 Thanks for helping out ;)
The Turf effect in this game causes a horrid memory leak that eventually hangs the video driver even if you're using an nvidia card.
@Squall-Leonhart : it turns out I've always played with NVIDIA tech disabled, but that might help others who encounter problems, so thanks for sharing. I tried enabling it just for fun and TurfEffects enabled looks miles better than disabled btw. What a shame ... Also, VXAO slows the game to a crawl on A770 (~15fps tops), and I'm conflicted about the ShadowLib implementation, it doesn't look any better. Sometimes it looks worse than when it's disabled.
Anyhow, here's a new update that basically invalides any and all of my previous findings : all dvxk versions I tried (including 1.10.3, 1.10.3-async, 2.0, 2.0-async, 2.1, 2.1-gplasync, 2.3, 2.3-gplasync) produce crashes, at various moments. Same as the native Intel Arc DX11 driver. I also tried the Special K library with it's suggested fixes for the game (disabling texture cache and lowering some engine threads's priority), no luck either.
This became very apparent on one particular boss fight (the Level 99 Adamantoise in Longwythe, for those who know the game). I simply could not get past this boss fight without a crash, sometimes mere minutes into the fight, no matter what driver or what setting I tried tweaking. I have the save file just before that particular fight for anyone who would like to reproduce, just ask, I'll upload it ASAP. It's a very simple but long battle, and the crash can randomly happen after anywhere between 2 to 30 minutes. I almost flipped my desk when it happened around the 50% HP mark ...
I was starting to think maybe my mobo/RAM combination is causing trouble, so I tried disabling DOCP (RAM was running at JEDEC 4800mhz) => same crash. Asus pushed several bios updates for my mobo already, and I installed them all, over time, no change (even yesterday, with a freshly published 2412 bios from 30th January 2024) => same crash. I've already used DDU multiple times and (re)installed multiples Intel Arc drivers over time for several months => same crash.
I quickly checked the steam community posts and noticed some people were complaining the game would crash on nvidia 3090 for example. I started to think the game engine was broken beyond repair.
The only thing that worked in the end (and that was a relief), was to swap out my A770 for my old 5700XT. I was then able to beat the adamantoise (took me an hour of non-stop button-mashing), and then go on for an extra 3 hours of uninterrupted gameplay.
I tried running the game in various debuggers and enabling various tracing options, the only useful information I would get out of this is either that the D3D11 device "drops", or that "igfxnd" stops responding (same thing really), either momentarily (pause in the game loop, which sometimes resumes), or indefinitely (the driver does not restart despite waiting for a long time, and after a few minutes of waiting I have to abort the game process). I might try tightening the microsoft TDR values and see if that somewhat avoids a crash. If I can get away with random pauses but no crash, I'd consider that a (albeit small) win. For recall, I had to loosen the TDR delays/threasholds to facilitate installing recent intel arc drivers in a single GPU configuration.
@IGCIT : are there any system events / wmi metrics I can log during gameplay to pinpoint what is happening at the moment of the crash ? I tried using the GPA tools but System Analyzer logs gigabytes of data per minute and I can't even open the traced data after that in Trace Analyzer (the program says the traced data is invalid, except I just captured it with System Analyzer that came in the same package ...). Graphics Monitor won't start the game, just like other debuggers I've tried. Maybe there's some hook prevention/anti-cheat mechanism getting in the way (some comments from the steam community hint at denuvo being used ...). VMMap crashes often, Process Monitor doesn't reveal anything useful AFAICT (it's more I/O oriented and has little GPU-related info), and DebugView revealed I had some Hyper-V activity going on in the background which I had no idea of. It's installed but I don't use it, so I removed it, maybe that will give some results, I'll report back at some point about this. I tried messing with RenderDoc (without understanding much of it) but capturing a "single frame" can yield from 500MB to 3GB of data depending on how many options I choose. A single frame, yeah, you read that right. Imagine a 15 minute gameplay ... I don't own a friggin' datacenter 🤣
Also (this could be interesting), I left my 5700XT in my system running over night, and came back to it this morning with 4101 events coming from the amd driver (amduw23g), spuriously generated out of nowhere during the night. At the same moment, Desktop Window Manager would produce a 9027 event "Desktop Window Manager has registered the session port".
The AMD driver greeted me with this bug report form (and I'm glad it did, otherwise I would have not noticed there had been a problem) :
Now I'm starting to wonder if my windows 10 installation is broken in some way, though I would really like to have some level of trace/metric to prove it before proceeding with a full system reinstall.
sorry, i don't have much to add to this, but, you can try to take a dx11 trace with https://github.com/apitrace/apitrace, if it works, the resulting file may be big in this case too, so it is probably best to save in a spot where a crash can occur more easily and try to capture multiple times to get the smallest
It is possible to share a savefile before the fight with the Level 99 Adamantoise in Longwythe?
New savegame : https://drive.proton.me/urls/BT4XBW9ZW8#hWRvlECpBmYc
is this the savefile?
It is possible to share a savefile before the fight with the Level 99 Adamantoise in Longwythe?
New savegame : https://drive.proton.me/urls/BT4XBW9ZW8#hWRvlECpBmYc
is this the savefile?
Thanks for your interest Arturo :) You're refering to an old save, here's the new one (single gameplay0.save file, grabbed from slot6 folder, which corresponds to slot7 in game) : gameplay0.zip
After loading the game save, it's pretty easy :
@IGCIT, @Arturo-Intel : I ended up trying starting a new game on a fresh Windows 11 install, same crash after about 30 minutes of gameplay. As usual, the crash happened after a few minutes into a random battle (easy hunter quest at the beginning of the game). So the Windows 10 vs 11, or even old vs fresh install issue is moot here, it's either driver-related or hardware-related.
Also, I found some Special K logs produced after a crash, and this appeared :
02/03/2024 22:11:06.571: DebugHelper Symbol Search Path......: ;C:\Users\***\Documents\My Mods\SpecialK
02/03/2024 22:11:06.571: Unable to load Special K Debug Symbols ('C:\Users\***\Documents\My Mods\SpecialK\SpecialK64.pdb'), crash log will not be accurate.
==================================================================================================
<+> System Memory Load....: 50%
# Current Working Set....: 5.874 GiB (Max Used: 5.888 GiB out of Total Physical: 31.219 GiB)
# Current Virtual Memory.: 13.509 GiB (Max Used: 13.514 GiB)
<+> GPU0 Load.............: 99.98% at 63.05°C
- Current VRAM In Use....: 6.237 GiB / Max Useable VRAM: 0.000 GiB
==================================================================================================
02/03/2024 22:17:51.537: Unhandled Top-Level Exception (c0000005):
-----------------------------------------------------------
[! Except !] << EXCEPTION_ACCESS_VIOLATION >>
-----------------------------------------------------------
[ Thread ] ~ Name.....: "LuminousTask1"
[ FaultMod ] # File.....: 'E:\SteamLibrary\steamapps\common\FINAL FANTASY XV\ffxv_s.exe'
[ FaultMod ] * RIP Addr.: ffxv_s.exe+000000000E8DA4D8h
[StackFrame] <-> Rip=00014e8da4d8h, Rsp=000020aafd70h, Rbp=00006bc7c280h
[StackFrame] >-< Rsi=000000000000h, Rdi=000052518050h
[ GP Reg ] rax: 0x000000000000 rbx: 0x00004e75bbd0
[ GP Reg ] rcx: 0x0000ffffffff rdx: 0x00039f706170
[ GP Reg ] r8: 0x000500000001 r9: 0xffffffff00000000
[ GP Reg ] r10: 0x000000000000 r11: 0x000020aafcf0
[ GP Reg ] r12: 0x000000000001 r13: 0x000042bc7ed0
[ GP Reg ] r14: 0x0000ac4e5fd8 r15: 0x0000ac4e5bd8
[ GP Flags ] EFlags: 0x00010202
-----------------------------------------------------------
ffxv_s.exe > NVSDK_NGX_D3D11_Shutdown
ffxv_s.exe > NVSDK_NGX_D3D11_Shutdown
ffxv_s.exe > NVSDK_NGX_D3D11_Shutdown
ffxv_s.exe > NVSDK_NGX_D3D11_Shutdown
ffxv_s.exe > NVSDK_NGX_D3D11_Shutdown
ucrtbase.dll > configthreadlocale
KERNEL32.DLL > BaseThreadInitThunk
ntdll.dll > RtlUserThreadStart
-----------------------------------------------------------
Full logs for inspection : ffxv_crash_specialk_logs_2024-02-03__23'17'51.zip
Also, using apitrace was pointless as simply running the game inside the tool made the game run at 10 seconds per frame (yes, you read that correctly). I couldn't even get to the game menu even after 15 minutes of waiting.
On the surface, this issue looks similar to this other issue : https://github.com/IGCIT/Intel-GPU-Community-Issue-Tracker-IGCIT/issues/695 , though I'm aware that a EXCEPTION_ACCESS_VIOLATION is rather generic and could be triggered by any code path. Seems to me that some code path in the driver isn't graceful at handling "exotic" input data or a particular edge case.
A hardware issue isn't ruled out yet, but then again, I'm only getting this specific type of crash in FFXV. Maybe the Luminous Engine used in FFXV is non-conformant to some extent, but I'd expect the driver to compensate for that to some degree, especially so many years after launch, and considering the game hasn't received a patch for the last 3 years at this point. I don't play Fortnite (which the user from issue 695 is playing) so I can't compare.
On the other hand : Quake II RTX => works great (although a little slow for my taste) Counter-Strike 2 => works great (also a DX11 title) Final Fantasy VII Remake => works great (went through the whole game without any issue, even when streaming via Moonlight server / Sunshine client to my TV) Trackmania with DXVK => works great (I haven't tried playing it without DXVK for a while, it used to not launch at all at the time ...) Metal Hellsinger => works great Serious Sam HD : The first encounter => works great Unreal Tournament 3 Black Edition => works great UT 2004 => works great Genshin Impact => works great
These are all games I managed to play multiple hours on end without any crashes on my A770 and with satisfactory performance, which is a testament both to the Arc product's line potential, and also somewhat to the stability of my system overall.
Intel released a new beta driver (5194), which I have just installed. I will try again with this one and report back.
... it crashed again, but after 4 hours of gameplay (yay \o/) using latest beta driver.
I encountered a total of four id 4101 "igfxnd stopped responding" during this session, 3 of which the driver recovered from gracefuly (after a short 1 second pause), and the last, where it gave up and just crashed the game. I can't tell if it was a lucky day or some sort of progress was made on the driver side, so I'll have to re-check tomorrow ^^ As usual, all occurences of that event are during fights. The last occurence was at the start of a fight, with multiple monsters spawning out of nowhere inside a dungeon.
Fights are heavily vertex-intensive if the graphs I posted above are to be believed (tons of VS threads being dispatched, so maybe a race condition ? a thread-pool exhaustion ? a bad pool synchronization ?), but also the camera sometimes moves inside of geometry due to spatial constraints. I mention this because this driver crash happened while my party was in a small room where 5 goblins spawned out of nowhere, and 4 party members were already standing. At that moment, the camera had erratic movement, probably trying to position itself somewhere useful, which might have been behind a wall. Might this be the edge case we're looking for ? I don't have a clue :/
Are you on windows 11? try disabling Control Flow Guard in defenders exploit controls.
Are you on windows 11? try disabling Control Flow Guard in defenders exploit controls.
Thanks for the suggestion. I tried this in Windows 10, no dice, still a crash. I will try in Windows 11 next, just in case MS is doing something a little different.
Edit: oh well, it was already disabled due to me applying the AtlasOS playbook on top of the Windows 11 install (something I didn't do on my Windows 10 install), so no luck there either ^^"
Just went through a full 12 hours of y-cruncher with the VST preset without any issues, so I'm certain my RAM is rock-stable (kinda reassuring, tbh I was starting to have my doubts). I'm running the vanilla DOCP DDR5-6000 profile that came with the RAM sticks (no tweaking at all). I used this preset simply because it's what BuildZoid uses over at his "Actually Hardcore Overclocking" youtube channel to test RAM.
Logs here : ycruncher_zen4_freak2fast4u_12hours.txt
Next I'll be trying with the default preset, which loads both the RAM and the CPU using several different number-crunching algos. I'll update this post tomorrow with the full log.
Edit : in the meantime, I DDUed the previous driver and installed 5330 beta => still not fixed :(
Edit 2: full y-cruncher logs with full CPU+RAM coverage, no issues after 12 hours straight : ycruncher_full_zen4_freak2fast4u_12hours.txt
I also tried using ProcessLasso, both to make sure the front window isn't hindered by background processes, and also to move both the GPU driver and the ffxv_s.exe away from CPU cores 0 and 1, since core 0 is overly loaded with many DPC and ISR calls (compared to the other cores, measured with LatencyMon).
I figured maybe some other driver is getting in the way, but guess what ? .... same crash :(
Edit : Tested VRAM for 1H with OCCT => No errors. Tested GPU core for 1H with OCCT (shader complexity 6) => no errors.
Welp, just ran the game, loaded a savegame that was at night, went to fight the daemons that spawned a few meters away, did a few eclipse assaults, and the driver stalled indefinitely, only mere seconds into gameplay. What the bananas is this ? oO
I was starting to consider this problem being linked to a memleak of some sorts, but I rebooted my PC just a few minutes prior to this event, so that hypothesis goes out the window ...
So all we know so far, after all this trial-and-error, is that the igfxnd driver stall is random, but only occurs during fights (that's the strongest clue we have concerning the trigger). It's also random to the point of being able to crash at literally ANY time, even within a few seconds of gameplay, within a few minutes after a system boot, so the memory isn't even bloated in any way. But also, it stalls 100% of the time during a fight, never elsewhere in the game.
This also makes any previous hypothesis about dxvk versions and/or Intel Arc driver versions impacting the gameplay duration moot as well ...
All I can say at this point is "lolwut ?".
I'm getting my hands on a non-steam release of the game as we speak for comparison, since several forum posts point towards the steam release of the game being broken to some degree, but then again, I'm having zero issues running the steam version of the game on an RX 5700XT, so it can't be just "the game being broken".
Same crash with latest 5333 WHQL drivers, 30 seconds (!) into a fight with a single enemy on screen (but many special effects). System was rebooted minutes prior, so no memory bloat possible : https://github.com/IGCIT/Intel-GPU-Community-Issue-Tracker-IGCIT/assets/3486369/2dad32bd-4e6b-4f12-bf92-ce62d00b6b4d
Corresponding saved game : gameplay0.zip
Enter the fort, talk to the floating computer, choose lvl 45 battle and fight until it crashes. It might take several tries to get it to crash, but no need to reload, just talk to the floating computer and try the fight again.
@freak2fast4u yes, I can confirm this. 101.5333 still have this behavior, will update internal case with this data. Thanks for the quick confirmation. (Indeed, you are 2fast4 me xDD)
Edit: thanks for the saved game file!
Is easier to reproduce with this settings:
3min into the fight and it crashes. https://youtu.be/RQ5wmDjFLUw
Will update internal case with the new data
--r2
Checklist [README]
Game / Application [Required]
Final Fantasy XV
Game Platform [Required]
Other game platform
No response
Processor / Processor Number [Required]
Ryzen 7800X3D
Graphic Card [Required]
Intel Arc A770 16GB LE
GPU Driver Version [Required]
31.0.101.5074
Rendering API [Required]
Windows Build Number [Required]
Other Windows build number
19045.3758
Intel System Support Utility report
igcit_ssu_20231213_ffxv_s_gfx_freeze_random_anon.txt
Description and steps to reproduce [Required]
Happens randomly during game play. OBS Capture here (the video is correct, the picture freezes on my screen during game play like it's shown) : https://www.youtube.com/watch?v=z0aOX3dAHsE
It feels very similar in behavior to this issue, which happened in CS:GO (DX9) : https://github.com/IGCIT/Intel-GPU-Community-Issue-Tracker-IGCIT/issues/444
Game graphic quality [Required]
Game resolution [Required]
2560x1440
Game VSync [Required]
Off
Game display mode [Required]
Detailed game settings [Required]
"Very high" (maximum) settings preset, Fullscreen, 60 FPS or 120 FPS limit (doesn't matter, freezes with both), 100% rendering quality, NVIDIA-specific technology disabled (hairworks etc.)
Device / Platform name
AM5 / B650
Crash dumps [Required, if applicable]
ffxv_s_evtx_and_dmp.zip
Save game
No response