PCSX2 Auto Test Suite

[gregory38]

A nice guy create a test suite: a https://github.com/unknownbrackets/ps2autotests

I create a nice script to run test (elf) tests/run_test.pl

Here the remaining item to do

• [x] add an help of the run_test.pl script
• [ ] fix bad test (priority are the crash/timeout)
• [x] add multiple configuration on the runner
• [x] find a way to blacklist bad test until there are fixed
• [ ] implement more test

• [x] add iop test on the runner

  find a way to run the test suite on buildbot (linux)

So who do what ? @ramapcsx2 @ssakash @refractionpcsx2 @turtleli

[refractionpcsx2]

i think @ramapcsx2 has already got some of these to run and highlighted some issues that need looking in to, he has a VM that can compile them and tested them on PCSX2 vs a TEST console.

[gregory38]

actually I provided him a basic test runner so it used to run test in cygwin. The new test runner requires perl (I think it can work on cygwin too) but linux might be easier. (I mean for the PCSX2 part).

However you might be interested to add some tests. Which make me remember that cotton wrote a vu testsuite too??

[refractionpcsx2]

I don't recall a test suite, i think some tests were done for testing VU oddities like multiple MUL's and ADD's in a row and getting the original answer or something ;p

[gregory38]

Is it possible to get/check all cotton post with an attachment on the forum?

[refractionpcsx2]

not for me, i dont have database access, that will be something for Bosit to do, if he's available.

[gregory38]

@bositman we search a dev post of cotton with some attachment to run VU tests. If you have any hint.

Edit: or am I drinking too much !

[gregory38]

@refractionpcsx2 Info on sysctrl test: Issue 1: infinte while loop in VifUnpackSSE_Dynarec::CompileRoutine

due to cl == wl == 0
so the else branch is always taken

Issue 2: Division by 0 because wl is 0

    if (!isFill) {
        // Accounting for skipping mode: Subtract the last skip cycle, since the skipped part of the run
        // shouldn't count as wrapped data.  Otherwise, a trailing skip can cause the emu to drop back
        // to the interpreter. -- Refraction (test with MGS3)
        uint skipSize  = (cl - wl) * 16;
        uint blocks    = v.block.num / wl;
        length += (blocks-1) * skipSize;
    }

[refractionpcsx2]

If wl is 0 then the unpack won't work anyway, in filling write there has to be a WL, else nothing will be written. I guess we could bottle out of it completely, but that should never be the case.

I think there's a hack in unpack actually for a similar scenario that Kingdom Hearts does and that stops it doing stupid things.

Edit: yeh here it is

if ((vifXRegs.cycle.wl == 0) && (vifXRegs.cycle.wl < vifXRegs.cycle.cl)) { //DevCon.WriteLn("Vif%d CL %d, WL %d Mode %x Mask %x Num %x", idx, vifXRegs.cycle.cl, vifXRegs.cycle.wl, vifXRegs.mode, vifXRegs.mask, (vifXRegs.code >> 16) & 0xff); vifX.cmd = 0; return; // Skipping write and 0 write-cycles, so do nothing! }

I guess that could be modified to just say if wl == 0

[gregory38]

Well don't know, but current code doesn't do what the PS2 does ;)

The others VIF issue is the programmation of the 2 bits mode. 3 is equivalent to 0. And I saw some basic mask.

[refractionpcsx2]

Yeh but the PS2 probably doesn't have any code doing divides to work stuff out ;) If games do anything that stupid they deserve to crash lol. I can fix it if you really want me to though, but a game that does it would be useful.

what are you referring to 2 bits but 3 is 0? that rings a bell but what you said didn't make any sense xD

[gregory38]

I miss you on IRC ;) We have a test that does it but I don't know if game does it (but you know). Edit: actually do we have game that freeze?

If you program the value 3 in the mode register the behavior of VIF will be the same as if you program the value 0.

[refractionpcsx2]

oooh the mode register! yes, 3 is ignored, that should be the case in the current code, unless cotton fluffed it up :P I'll look in to it though.

[gregory38]

Current behavior seems to be the offset mode (so 1).

[refractionpcsx2]

virtual bool IsUnmaskedOp() const{ return !doMode && !doMask; }

I bet that's why, he's assumed if the mode isn't 0 it uses mode 1 and there is 1 line to do the special handling of mode 2, but nothing for 3 (that i can see)

[refractionpcsx2]

We could put handling for this in to VifUnpackSSE_Dynarec::VifUnpackSSEDynarec(const nVifStruct& vif, const nVifBlock& vifBlock_)

thats in newVif_Dynarec.cpp

what do you say? its either that or we put it after this line of the same file line 195

doMode = (upkNum == 0xf) ? 0 : doMode; // V4_5 has no mode feature.

[gregory38]

I already quickly tried the line 195 but it isn't enough. Vif code is black magic for me.

[refractionpcsx2]

what did you do? i presume you did

if(doMode == 3) doMode = 0; ?

if not then you should have xD

[turtleli]

I assume the buildbot stuff is for me ;)

[gregory38]

Here the test diff with your patch.

-----------------------------------------------------------------------
/home/gregory/playstation/emulateur/pcsx2_merge/bin/PCSX2 --elf /home/gregory/project/ps2autotests/tests/dma/vif/mode.elf --cfgpath=/home/gregory/project/ps2autotests/tests/dma/vif/mode_cfg

--- /home/gregory/project/ps2autotests/tests/dma/vif/mode.PCSX2.out 2016-01-19 09:34:40.719208662 +0100
+++ /home/gregory/project/ps2autotests/tests/dma/vif/mode.expected  2015-12-22 13:51:51.142960920 +0100
@@ -46,13 +46,13 @@
   vumem(10): 00000004 - 00000005 - 00000006 - 00000007
   vumem(20): 00000008 - 00000009 - 0000000a - 0000000b
   vumem(30): 0000000c - 0000000d - 0000000e - 0000000f
-  row: 00001000 - 00001000 - 00001000 - 00001000
+  row: 0000000c - 0000000d - 0000000e - 0000000f

 mode 3 (mask diagonal):
   vumem(00): 00001000 - 00000001 - 00000002 - 00000003
-  vumem(10): 00000004 - 00001000 - 00000006 - 00000007
-  vumem(20): 00000008 - 00000009 - 00001000 - 0000000b
-  vumem(30): 0000000c - 0000000d - 0000000e - 00001000
-  row: 00001000 - 00001000 - 00001000 - 00001000
+  vumem(10): 00000004 - 00000001 - 00000006 - 00000007
+  vumem(20): 00000008 - 00000009 - 00000006 - 0000000b
+  vumem(30): 0000000c - 0000000d - 0000000e - 0000000b
+  row: 0000000c - 0000000d - 0000000e - 0000000b

 -- TEST END
-----------------------------------------------------------------------

I assume the buildbot stuff is for me ;)

Yes :+1: Honestly I don't know if we can do it but it will be interesting to do a summary of the issue so they can fixed one day. Note test requires a default config. The plan was to put something a default ini dir on the tests directory.

Command example (with only pass test)

./tests/run_test.pl --suite ~/project/ps2autotests/tests/cpu/ee_simd  --exe ./bin/PCSX2 --cfg bin/inis --cpu 2

[refractionpcsx2]

I don't understand that diff? even the row's are different..

[gregory38]

Oh sorry. The line starting with - are the from PCSX2 and the + are the expected. You can get the test and the expected here https://github.com/unknownbrackets/ps2autotests (test is stmod). It might be easier to understand what going on. (the while loop issue is stcycl test).

I think it would be easier for you to be able to run the test on your side. I think you can directly run the .elf on windows. @ramapcsx2 normally Perl could work on windows. I think it would be easier for everyone if we manage to get the script working on windows rather than relying on VM.

[refractionpcsx2]

ok ill give it a go tomorrow, not sure ill have time tonight (i'm not home until late tonight)

[gregory38]

no worry we have plenty of time :)

[refractionpcsx2]

hmm just looking at the original "expected" file, mode 3 isn't seen as mode 0 at all, it's close but the row data gets overwritten.

every test sets the row to 0x1000 for each vector, however in the mode 3 tests the last set of vectors written to the VU overwrites the row contents (from the looks of things) it would be interesting to see how that unfolds with an extended set of data, if it's only ever the last written VU data that gets written back or if its the last of the first block (ok less likely, but stranger things have happened)

[gregory38]

Ah. I better understand why the fix doesn't work :p (time to flee now ;) )

Feel free to extend the test. It would be nice if we can have an industrial test solution. It might help to fix corner case, but it will help a lots in case of code update and regression.

[refractionpcsx2]

Well I would like to implement my unpack tests properly, I have got ELF's from when i did it before in a very "manual" method :P but would be good to actually use the SDK lol

[ramapcsx2]

Yeah, we still have many old tests, many with c code available. I'll collect them and give them to @gregory38 :) Getting the tester to run on Windows would certainly be preferable. I'll look into that.

[gregory38]

Hum, I might need to port system call to perl equivalent. Otherwise I'm afraid it won't work but it must be doable (annoying but doable).

[refractionpcsx2]

I need to learn how to use my build environment that rama gave me so I can actually build things properly :P I have got a cygwin setup with the open SDK on it (probably an archaic version now) which I used previously, but never actually used the SDK, I did every thing with arrays and register writes lol

[ramapcsx2]

@refractionpcsx2: It's embarrassingly simple with the vm, simpler than the old crappy msys way we used to do ;) Often you don't have to recompile the tests though, simple use the provided elfs. @gregory38 what am I doing wrong? https://i.imgur.com/1y1TdeA.jpg

[refractionpcsx2]

@ramapcsx2 I know you don't have to recompile them, I meant for compiling my tests.

[gregory38]

what is your env? cygwin? I added on option --test to select a subset of test. So I suggest to use --test alu to reduce the number of test. Then, is PCSX2 fired? If yes, check the content of CFG directory. For example alu_CFG. I copy the ini and update the configuration. You need to check that print iop/ee are enabled.

[refractionpcsx2]

@gregory38 I have several lol, got 2 VM's with linux on, not sure what it's using to build it and I have an old cygwin setup under windows for compiling as well which is what i used to use when I was making tests.

[gregory38]

I mean rama ;)

[refractionpcsx2]

oh, sorry for butting in :p you should have tagged xD

[gregory38]

And I need to setup the sdk too to on my side. It will likely very easy for me because I'm on linux already ;)

[refractionpcsx2]

its not as straight forward as copying it, you have to mess around a bit :P

[gregory38]

feel like standard linux stuff.

@ramapcsx2 did you see the question above

[ramapcsx2]

Yep :p

[ADormant]

@unknownbrackets Are GS tests possible?

[refractionpcsx2]

Any tests will be possible.

[refractionpcsx2]

So, somehow i have to convert this to something sensible and redo all the tests.. lol http://pastebin.com/xdvvpB87

It was as painful to code as it looks.

[unknownbrackets]

FWIW -

1. We use Python in PPSSPP. I've found it to be more portable and less hassle than Perl, but of course they both work. You can look at test.py in the ppsspp repo. It even has matching subsets of tests, etc. and handles timeouts.
2. To compile tests you need a network adapter, ps2link, and ps2sdk. I think I had to build ps2sdk from the latest git but it wasn't much more than that. I run it all on Windows, no VM needed.
3. Graphics tests will require offscreen rendering or showing the window and rendering. We do that with PPSSPP (offscreen), but we also blit the data to RAM in the PSP tests so we can check the data directly in PPSSPP (otherwise it doesn't end up reading from pixel reads.) There might be some tricks like that needed for PS2 GS tests also.
4. You can check the ppsspp repo for running unit tests on Travis. Also, we run them in Teamcity. Happy to share the scripts.

I tried to set up a clean framework for writing VIF tests, but let me know if you have any feedback. There's tons of room for more tests, I've just been more messing with PSP stuff more recently...

-[Unknown]

[gregory38]

Perl is much more powerful than Python. Because I know Perl very well ;)

GS is less important because bugs are due to not-emulated feature on purpose. However it could be still useful as non-regression checker.

[ADormant]

GS is less important because bugs are due to not-emulated feature on purpose.

What is this feature and why you are not emulating it on purpose?

[refractionpcsx2]

What is this feature and why you are not emulating it on purpose?

It's features like post processing and depth, mainly because of the design of the texture cache it could not support depth and the post processing effects until DX11 and OGL 4.5 weren't possible to do as they are done very differently from PC games, in some instances they did funny things with writing back to existing textures, again something the texture cache didn't support until recently, so they were being skipped, or left the game looking a mess.

[gregory38]

We need also to emulate the buffer of the GS memory. After a write, you can either read the written value or the previous older value. Even the SW render doesn't emulate it.

GS is integer, GPU are float. We could use integer but -20%. We need to disable HW interpolation and do it manually, again slower.

In short speed, speed, speed. And bit accurate image will be very painful to test.

[ADormant]

@refractionpcsx2 Are post-processing effects already fully emulated? So that leaves mipmapping? Integers aren't that big of a speed impact probably - Dolphin emulates them easily.

[gregory38]

Please let's me correct. Post processing effect are barely emulated now. It misses various effects (depth/texture shuffle etc..).

[refractionpcsx2]

Yeh post processing is a lot better than it was but far from being complete and accurately emulated. Depth still needs work, but aside from fixing graphical stuff, this will fix the camera in the fatal frame games too, which will make people happy.

MipMapping is a distant goal which may not be reached for many years, unless we can come up with a cunning way around it. It would be nice if we could essentially force the game to use the biggest textures available, there'd be shimmering but at least it would look right :P

Dolphin may emulate them easily but it really depends how the effects are done. The graphics unit on the gamecube makes a lot more sense (far superior too) and in turn is probably a bit easier to emulate, the whole ps2 is just a mess of technicalities.