Stuntman - FPU rounding errors for AI pathing

[Souzooka]

PCSX2 version: PCSX2 1.5.0-dev3032

• Notes: In Stuntman, AI paths are slightly erroneous due to FPU rounding issues; the game is unable to be finished due rounding issues. This can be witnessed in any level with a car the player is made to follow. Specifically, in the fourth level of the game, the player follows a lead car using a police car. On the "Chop/Zero" FPU rounding mode, the lead car fails to make a corner in the level, softlocking the level. Changing the rounding mode to "Nearest" allows the car to make it past this corner, but it fails the very next turn.

How to reproduce the issue: Start a new career in Stuntman, finish the first three levels, then attempt to finish the fourth level.

Last known version to work: While the game is marked playable, there is no evidence that the game was ever checked that it was able to be completed. The game is pretty resource-intensive, and it is more likely that the first level was tested at most by most testers.

Attached is a picture of the lead car failing to make a turn on the "Chop/Zero" setting.

[Souzooka]

I attempted to try this PR and while the AI gets even further on interpreter, he still fails a turn later.

[ghost]

Try to set also VU clamping/rounding. This game use both FPU, and VU to calculate physics there. Problematic calculations results are from in game functions dyForwardDynamics, and dyStepRunge1 to dyStepRunge5 (0x211E60 to 0x2127AC for US release).

[LoStraniero91]

This also happens in Driv3r (and probably Driver: Parallel Lines). Could be the same issue since all of them were made by Reflections.

[refractionpcsx2]

I know it probably isn't fixed, but can this be tested on a recent master to see if it's any better?

[ghost]

Yes, issue is still present on latest master.

Edit: I made few tests, and looks like game don't like hardcoded DIV/SQRT/RSQRT nearest rounding on FPU. Although removing it help only little bit, you still can't finish level that way. That's for recompiler, interpreter seems to be broken even more on that game.

Here are settings, and explanation where you can get with them.

FPU / VU (All test done with removed hardcoded FPU nearest hack) Negative / Zero Best result, car crash at the wall in the end.

Nearest / Zero Result like on Souzooka picture.

Positive / Zero Crash earlier, car goes too much on right side.

Negative / Positive or Nearest Close to first picture, but crash to wall at left little bit earlier than on VU Zero. There is no way to test properly VU negative rounding because that mess VU1, and picture. But setting it for moment make car path worse when we set it back. Tried also negative div hack, seems to have no effect. So that suggest sqrt/rsqrt nearest rounding make it bad.

Edit2: Looks like i missed that pcsx2 use different file for doubles, and didn't removed hack there previously. So results stay as above, but now we can get to "best result" even on Zero/Zero rounding. Still no way to finish stage.

[arienap]

I did some more testing with the PAL version of this game and compared the behavior of the AI car to an actual PS2. I used a later level (Woopin and a hollerin) and by setting the EE rounding to nearest I get the AI to behave closest to actual PS2 AI. EE rounding to zero, positive or negative make the AI drift further sooner from the game on PS2. For all other settings (VU rounding and EE/VU clipping) I could not see the effect on the AI car. Tested with v1.7.0-dev-546-g64010cf79 (latest GIT build). Game plays nicer on PCSX2 than on my PS2+TV, I think because of the better analog sticks of the Xbox360 pad combined with less input lag.

[ghost]

areinap, if you can send me a save of your game to the Woopin and a hollerin mission as well as a savestate before the car get stuck I would appreciate it, thanks.

[arienap]

Stuntman_debug.zip I have uploaded a memory card with the 100% completed career from my PS2. I have only completed the PAL version and the debugging was also done with the PAL version. You can try the Woopin mission (nr.2 corkscrew) by loading the game "Arie" from the main menu and then go into the filmography in the career menu. For your convenience, I attached a save state of doing just exactly that. The save state is at the beginning of the mission, the cars path is affected right from the start. By changing the EE FPU settings you must restart to see the full effect (the path of the cars drifts earlier / later).

[ghost]

Thanks a lot for uploading booth the memorycard and save state, I will look into the game.

[ghost]

I can make the car almost right with 4 settings, EE Rounding to negative - EE Clamping to none - VU Rounding to nearest and VU clamping to none. It will however stops in the middle of a straight road.

[arienap]

No setting the EE rounding to negative makes the car go further in the level, but its behavior is much different compared to the path it takes on the actual PS2. In the level you are testing, the car on the PS2 goes between the van and the car after the right hand corner. With EE rounding to negative it makes the car go left of both.

I am pretty sure it is related to the FPU rounding of the recompiled EE code. The VU rounding might also contribute a bit, but from trying the settings I could not see the effect. Would it make sense to make some homebrew code to exercise the DIV/SQRT/RSQRT instructions and compare the outcome of the actual PS2 HW versus the recompiled code?

[refractionpcsx2]

@arienap Did you try my FPU rounding PR when I had it open? DIV/SQRT/RSQRT are forced to "nearest" rounding by default in PCSX2, that PR removed that so you could change the rounding for the EE and it would actually affect it.

[arienap]

You mean this PR: https://github.com/PCSX2/pcsx2/pull/3816 ? I have not set up the toolchain to build PCSX2 myself, so I was using the available dev builds from master (PCSX2 v1.7.0-dev-546-g64010cf79 in this case). I still see significantly different behavior when changing EE settings, so I guess there must be more to it than just the DIV/SQRT/RSQRT instructions. I am happy to do more testing, but I need some time, so I can figure out how to build PCSX2 myself.

[refractionpcsx2]

Each PR has its own builds at the bottom, however when it gets closed they get removed so of course they're gone now.

I guess, i can make you a build temporarily.

[refractionpcsx2]

pcsx2-noforceround.zip Here, try this, you should be able to adjust the EE rounding modes with this and have more impact. Chop/Zero will actually be Chop/Zero for those commands now, so be sure to try that too.

Edit: Also be sure to try all rounding modes with Full clamping as well, as full clamping actually changes the FPU to use doubles which is completely separate code for handling FPU.

[arienap]

I tested the build, thanks for providing this. I tested with multiple levels (stunt drivers, ice river drive in Switzerland). The AI drivers are driven by the game in open loop, giving throttle and steering inputs. The EE rounding mode messes with the physics which in turn mess up the AI drivers path. It is a little bit misleading to judge the correctness of the emulation by the progress of the AI driver, as the AI driver has very little margin. In some cases the even more inaccurate emulation allows for the AI driver to progress further. Especially in the Switzerland level, the most accurate mode lets the AI get stuck the earliest. I tried many different settings for the EE clamping / VU rounding / VU clamping modes, but I could only really detect changes with the EE rounding mode.

All four EE rounding modes behave differently, but by comparing them to the actual PS2 I would rate them the following (accurate to least accurate):

1. Nearest
2. Negative
3. Chop/Zero
4. Positive

The EE rounding positive gives more errors than just the AI driver, you feel more grip (I can achieve faster lap times in the speed tests with EE rounding to positive) and the sound gets messed up.

[refractionpcsx2]

Okay well the rounding that is forced is forced to nearest, so I guess we're already doing the best we can. Apparently the AI on the PS2 isn't spectacular anyway, but I assume it doesn't get stuck

[ghost]

Actually VU clamping to none improves the ia somewhat too, but it get stuck one turn on the another still.

[arienap]

OK I moved to the NTSC version for some more testing. The NTSC version seems affected more by the emulator compared to the PAL version. This may be due to the physics running at a higher frequency (60Hz vs 50Hz). The first chase in the "Live twice for tomorrow" seems perfect for testing as the AI does not slide around as much, so it can drive for much longer without crashing. I do see an improvement by using the EE clamping mode "Full" compared to normal, so this setting is affecting the physics code albeit much less compared to the round mode. I will do an extensive test with all combination of settings later and provide the data.

[ghost]

Hello, 3y old issue, but the wiki states that the pathing is "very close to a real PS2" which is wrong, as on the 2nd level of the 2nd movie, the ai breaks and stops you from completing the mission https://youtu.be/BGNTgZ9BrrI Tested on PCSX2 1.7.2352

Round mode and clamping mode for the EE, FPU and VUs are all on default. settings

[SoapyMan]

Did tests on the 1.7.3069 AVX2 and when EE rounding mode set to Nearest on Whoopin and Hollerin chase - the getaway doesn't hit the pickup truck just like on hardware (reference: https://youtu.be/Goda4uRzrH8?t=94) but after that it hits the rear bumper at the right side when passing the van and hits the fence.

[ColeTrickIe]

Unfortuneately still there in 1.7.3327

[BenoitAdam94]

Guy here couldn't finish the game on PCSX2 : https://www.youtube.com/watch?v=5mbT7HRa7YQ

[Shoegzer]

Right, here is the spot where ColourShed encounters the issue with PCSX2. I haven't tested Driv3r lately but it's likely still the same given that it also abuses PS2 floating-point inaccuracy as reported above and here. The fact that it's also a Reflections game and probably using the same engine is a big hint too.

I'm guessing this is one of the hardest problems remaining to solve given that both titles (especially Driv3r) are arguably the highest profile games still not playable according to the compat list. It's not surprising given the undocumented complexities of the EE, though at least a happy accident that resolves this would be nice to see down the road.

[BenoitAdam94]

Right, here is the spot where ColourShed encounters the issue with PCSX2. I haven't tested Driv3r lately but it's likely still the same given that it also abuses PS2 floating-point inaccuracy as reported above and here. The fact that it's also a Reflections game and probably using the same engine is a big hint too.

I'm guessing this is one of the hardest problems remaining to solve given that both titles (especially Driv3r) are arguably the highest profile games still not playable according to the compat list. It's not surprising given the undocumented complexities of the EE, though at least a happy accident that resolves this would be nice to see down the road.

Driv3r has a PC version anyway, that is a bit superior and less buggy. But for Stuntman there is no alternative. This game really need the PCSX2 dev attention.

This probably will need a hack code only applyable for this game.

[Shoegzer]

Maybe from a game player's perspective, but from a preservation standpoint it misses the point. These games should be properly preserved whether or not they are on PC.

[arienap]

I think fixing this issue is very difficult and time consuming. I am pretty sure the issue is due to the hardware floating point inside the PS2 not following the IEEE 754 standard. A very similar issue found in emulating the PSP floating point is found here: https://github.com/hrydgard/ppsspp/issues/2990 It would not be possible to emulate the PS2 exactly by recompilation, since every IEEE 754 rounding modes supported by modern CPU instruction sets will not produce results exactly like a PS2 would. The PS2 hardware has its own quirky, not well documented, rounding. And even if we fix the rounding, there might be other FPU differences causing this game to break.

Further insight in the game can be had by analyzing an early "exclusive" demo with debug symbols. The "AI" in this game is nothing more than an actual physics object controlled by pre-recorded control inputs. Even if you find the bits of code which suffer from the inaccuracy of the floating point unit and patch them, you will still be stuck with non working AI. The AI paths were created on a PS2, so only with the same inaccuracies in the physics calculations, it will follow the intended path. So I am not sure if a game specific hack would fix the AI for this title.

I guess the only way forward would be to replicate the floating point inaccuracies of the actual PS2 hardware. That would probably start with writing a custom piece of code and running it on PS2 hardware to understand the deviations from IEEE754. Then it would need specific patches for some instructions and/or data combinations. Not sure if anyone want to sink time into such a project and how feasible it would be to support these inacuraccies into the emulation. Maybe the devs can comment whether such thing would be even feasible.

[refractionpcsx2]

yes, it's caused by float inaccuracies, usually modifying the rounding modes for the EE changes the behaviour, kind of as expected.

The solution going forward would probably to be some sort of soft float implementation, the only reason it hasn't been done yet is because it's non-trivial and would likely be very slow on a lot of systems. I believe Sony uses it in their emulator, but they basically have per game hacks to only enable it in specific situations to try and keep the load down, and this is further stuff PCSX2 can't currently do for things like this, so that would also have to be programmed, unless we just had "Softfloat YOLO!" (slow) mode for it.. lol

[stenzek]

We would also have to know how the EE rounds for several instructions, even for softfloat, which is currently unknown. AFAIK add/sub is fairly accurate, but mul/div/etc are still unknown.

[BenoitAdam94]

Fact :

REDRIVER 2 has custom chase added

Maybe @SoapyMan can help here ?

[arienap]

I did some digging and found an IEEE paper called The vector floating-point unit in a synergistic processor element of a CELL processor This document clearly describes the implementation of the single precision floating point units found in the SPE's of the PS3 CELL BE. Now the interesting part of this paper is the mention of the Emotion Engine's FPU:

The FPU of the Emotion Engine of the PlayStation 2, only supports truncation rounding and normalized numbers

Although an evolution of the EE's FPU, the CELL BE SPU FPU does not behave exactly like the PS2 FPU. The paper also refers to the following paper which goes into detail of the PS2 FPU: 2.44-GFLOPS 300-MHz floating-point vector-processing unit for high-performance 3D graphics computing Unfortunately, I cannot find that specific paper in public domain. Maybe someone with an academic background could help us get that paper. It might provide some insight in the limitations of the PS2 FPUs.

[mirh]

https://ieeexplore.ieee.org/document/848212 But I don't think there was really much mystery left into the operational architecture of the cpu.

[JulianWgs]

From the paper:

The floating-point execution units support IEEE-754 compat- ible 32-bit single precision format. A “not a number” (NAN), an infinite number, and a denormalized number are not supported. This is because an execution with no exception is more impor- tant for the graphics applications than getting a mathematically accurate result. In the floating-point execution units, a NAN and an infinite number of the IEEE-754 format are treated as normal- ized numbers, and a denormalized number is treated as a zero. These execution units do not support IEEE-754 rounding be- cause its complexity causes a long latency. These units calcu- late with guard bits; then the difference between the result of these units and the result that is rounded by IEEE zero direction rounding is only up to one least significant unit of the fraction. We judged this precision accurate enough for the graphics ap- plications.

You can find the paper on website which removes all barriers in the way of science.

Hope that helps :)

[Goatman13]

Game need accurate ADD/SUB for COP2. At least it looks like accurate VADD.xyzw will be first step. I tested rounding per single instruction, and only difference in path is when i change VADD rounding. I also tested some hacks, like AND all "to" vectors with 0xFFFFFFFE in VADD opcode, and result is even better than with nearest rounding, but still wrong. So is little bit guessing, but it seems that VU also need single guard bit add/sub. But this will be pain to implement and can be really slow for all vectors.

Small note if someone want to start doing this. microVU_Misc.inl have code for single vector almost ready, there is just missing comparison to 0 at line 382. While this is good place to start, code for all vectors at once need a lot more work.

[refractionpcsx2]

Yeah we need a good system to allow full accurate float emulation because we are missing that guard bit nonsense, but this isn't an isolated problem, there's a bunch of games that need it, and it's probably split over add, sub, mul and div.

Does the Add/Sub gamefix (which enables the "accurate" one on the VU's not kick in with COP2?

[Goatman13]

Current add/sub gamefix is really stripped version of accurate add only, not even cover all possible cases for add. Minimum to make triace games happy. From my limited testing Stuntman need that fix for VADD.xyzw, which is done with ADDPS, so gamefix don't even trigger for that. Also current "Add/Sub" fix work only on ADDi/VADDi.

[refractionpcsx2]

Yeah, it's not great what's there, but I dread to think how much code we'd need to make it happy.

[Shoegzer]

Can someone with the appropriate game saves please test Stuntman and Driv3r per https://github.com/PCSX2/pcsx2/pull/7707#issuecomment-1366061561? Would be so great if the new split-VU rounding/clamping feature allowed for a decent workaround to this issue.

[Sukotto-1999]

Regarding this issue, September 2016, a user named FlatOut on the PCSX2 fourms, Found a slightly-BIG issue that completely makes DRIV3R unplayable! THIS is the most affected game on PCSX2 developed by Ubisoft Reflections. The police car always crash into a different location. Issues may not be fixable at the moment.

[refractionpcsx2]

We're fully aware.

[Shoegzer]

Yeah, I guess I should have been more specific about what we'd be testing for here so it's good to have the clarification.

[iyudar]

For what it's worth, I remember using the director mode over anything else in DRIV3R over a decade ago, and I occasionally remember the recording of your pathing randomly messing up causing this identical bug that appears in Stuntman, on ORIGINAL PS2 hardware. I would do some cool cop chase, go to edit the scene and find out a lamp post i drove very close to in-game, ends up being in the way of my driving path in the recorded clip and the scripting illusion completely fails.

Whatever caused the error in the code on PCSX2 doesn't seem necessarily exclusive in an emulator, it's just more prominent or consistent on an emulator. It's still present on a regular PS2; what causes it there?

[sittingduck77]

I know nothing about programming. But could there be something to gain by looking at how Stuntman (and Driv3r) runs on the PS3?

[refractionpcsx2]

Probably not, the PS3 emulator has some measure of soft float emulation, which is the solution we've previously mentioned.

[mirh]

Both run like crap (even on ps4). To be super fair, the other two built-in software emulators might have different results but I wouldn't really try to grasp at such straws (even because, even in the most rosy situation, you'd still eventually have to figure out what's the magic).

[sittingduck77]

My thinking was, that the PS3 was pretty much completely hacked with CFW and so that it would be easier to figure out the magic. But maybe it's not THAT open.

[stenzek]

Legally, we can't use anything from the PS3/PS4 emus. Had this discussion several times, but even looking at the emu itself is questionable, because it's presumably protected from reverse engineering by access control.

[JulianWgs]

I've stumbled across this article today (good and short read by the way). @F0bes already is a contributor to PCSX2, but I dont know whether they is aware of this issue. May be they can offer some advice on how to tackle this.

PS: Please don't pressure any PCSX2 contributor into doing something on this issue. They are doing this in their free time and the software is provided "as is".

[refractionpcsx2]

Fobes works very closely with us, helps out quite often, but I'm pretty sure this is a little out of his reach, as it is the rest of us.

we know what we need, we need a soft float implementation, fixing 1*N isn't going to make Stuntman magically work, unfortunately.

[terremoth]

This also happens in Driv3r (and probably Driver: Parallel Lines). Could be the same issue since all of them were made by Reflections.

Yes, it HAPPENS in Driv3r indeed!