cputest_frsp fails on real hardware

[Pokechu22]

On my Wii, running the frsp test results in this output:

Hello world!
Subtest 11 failed in ../cputest/frsp.cpp on line 34: frsp(0xfff7ffffffffffff, NI=0):
     got 0xffffffffe0000000
expected 0xfff7ffffe0000000
Test 1 failed (12 subtests, 1 failures)
Shutting down...

On Dolphin (with both the x64 JIT and the cached interpreter), I get this output:

Hello world!
Subtest 7 failed in ../cputest/frsp.cpp on line 34: frsp(0x380fffffffffffff, NI=1):
     got 0x3810000000000000
expected 0x0000000000000000
Subtest 11 failed in ../cputest/frsp.cpp on line 34: frsp(0xfff7ffffffffffff, NI=0):
     got 0xffffffffe0000000
expected 0xfff7ffffe0000000
Test 1 failed (12 subtests, 2 failures)

My conclusion is that the test incorrectly handles SNaN:

https://github.com/dolphin-emu/hwtests/blob/328faf95cefe9f1d2e8cb0800f4d92c2e21a2a66/cputest/frsp.cpp#L22

though Dolphin is incorrectly handling the other case (I previously linked the wrong line (the one for smallest positive single normal); this is the correct one):

https://github.com/dolphin-emu/hwtests/blob/328faf95cefe9f1d2e8cb0800f4d92c2e21a2a66/cputest/frsp.cpp#L18

[JosJuice]

Indeed, according to the PEM, bit 12 shall be set to 1 if the input is an SNaN (i.e. SNaN gets converted into QNan).

Not sure why Dolphin is failing on the smallest single normal, though.

[JosJuice]

Turns out this was fixed in a still open PR from 2014... PR #23

As for the case where Dolphin is wrong, this should be filed on the Dolphin issue tracker. Presumably the hardware is flushing subnormals to zero before rounding instead of after. This would require a small performance hit to be accurately emulated in Dolphin.

[JosJuice]

Console failure fixed in https://github.com/dolphin-emu/hwtests/pull/23, Dolphin failure fixed for interpreter in https://github.com/dolphin-emu/dolphin/pull/10959.