GTE opcodes are not unpatched on exception return

What steps will reproduce the problem?
1. Play Spyro 3 (NTSC tested, but all versions probably fail similarly)

What is the expected output? What do you see instead?
One does not expect the anticrack checks to fire, but they do, causing the game 
to be unstable/not completable.

What version of the product are you using? On what operating system?
SVN trunk (r722) was tested, but the issue probably exists on v2.0.2 (and 
other/all?) releases too.

Please provide any additional information below.
PSXjin, in code that appears to be inherited from PCSX, has an ugly hack in an 
attempt to make exceptions that are raised when a GTE op is scheduled to be 
executed work properly.

On real PSX hardware, when such a situation occurs, the GTE op is performed in 
parallel with the exception handler being invoked. The BIOS then contains code 
to check whether the EPC points to a GTE op when the exception handler is 
complete. If it does, it skips it so that it is not executed twice, which would 
lead to slowdown, or worse, incorrect results in the GTE registers (if the op's 
output is used in the input as well). See [1].

PSXjin does not execute GTE ops in parallel with (ie. before) the exception 
handler is invoked, and hence needs to hack around the BIOS wanting to skip the 
GTE op on exception return. It does this by munging the GTE opcode in-memory so 
that the BIOS does not recognize it as a GTE op, and so that it returns to it 
again for execution post-invocation like any other op would.

In Spyro 3 there are two elements to its copy protection[2]. The first is a 
standard anti-modchip protection, which attempts to detect the presence of a 
modchip by looking for the CDROM controller picking up SCEx strings on sectors 
where they would not normally be found.

The second is a thorough checksumming of the game's binaries in-memory as the 
game is running. If the checksums do not match their expected values, the 
various routines scattered throughout the code will fire, which impede the 
player by removing progress, or making it impossible to proceed.

PSXjin's runtime patching of the game's binaries is causing these second 
checksums to be incorrect, leading to the anticrack routines firing.

There are two possible fixes to this problem.

The first is to correctly emulate the PSX and perform GTE ops before invoking 
the BIOS, and not patching the memory.

Likelyhood of adversely affecting existing games (desyncs, not working at all, 
etc.): Medium
Likelyhood of helping other games (that may be similarly affected): Low; it is 
unlikely many other games notice the GTE op patching.

Since this is not as good an option as the second, a patch is not provided, 
however it should be easy to do if necessary.

The second is to simply unpatch the patched GTE opcodes when the exception 
handler returns, ie. on RFE opcodes.

Likelyhood of adversely affecting existing games: Tiny
Likelyhood of helping other games: Low

Since this is the better of the two options, a patch is provided (against 
current SVN trunk, r722) which implements this fix.

It requires storing the patch information at runtime, with it being stored in 
savestates in a backwards-compatible fashion if ones are made during gameplay.

Thanks to Nitrofski (twitch.tv/nitrofski) for assisting in testing the patch!

References:
[1] http://nocash.emubase.de/psx-spx.htm#cop0exceptionhandling, "Interrupts vs 
GTE Commands"
[2] http://www.gamasutra.com/view/feature/131439/keeping_the_pirates_at_bay.php
Original issue reported on code.google.com by tiber...@gmail.com on 6 Aug 2013 at 6:27
Attachments:
unpatchgteops.diff
nitrofski / psxjin

GTE opcodes are not unpatched on exception return #39
