Need nightly automated video plugin regression test

[GoogleCodeExporter]

In an effort to insure the quality of the Mupen64Plus video plugins and
avoid time wasting regressions, I would like to set up a nightly automated
video plugin regression test.  This system will do a nightly checkout and
build of the SVN trunk, reporting any warnings or errors, followed by a
series of tests with each video plugin over a broad range of games.  The
report will be automatically emailed each night to any developer who wishes
- at a minimum the people working on the plugins.

I will add 2 new small features to Mupen64Plus itself to support this
regression test.  First, I'll add a frame counter and print the frame
number to the console each time the user takes a screenshot.  Second, I'll
add a new command-line option which takes a list of frames, and the
emulator will automatically record screenshots at those frame numbers, then
close down after the last frame is saved.

This system also depends upon the completion of issue #60.

Original issue reported on code.google.com by richard...@gmail.com on 19 Apr 2008 at 4:29

[GoogleCodeExporter]

Just a bug to point out about that would effect this. Some video plugins (I've 
only 
seem this in Rice personally) don't properly set up the full video state on 
resume 
from a snapshot. 

To see this yourself:

1) Load Mario64.
2) Play until you get to Level 1, then pause and see the level status / quit 
text.
3) Save a snapshot.
4) End emulation, reload Mario64.
5) Load snapshot immediately after seeing Mario64 text - the level status / 
quit 
text will not appear.

6) End emulation, reload Mario64.
7) Load snapshot after getting to the select saved game text - the level status 
/ 
quit text will appear.

I'm assuming that the automated tested would load a ROM and a savestate.

Original comment by sknau...@wesleyan.edu on 19 Apr 2008 at 5:47

[GoogleCodeExporter]

I'm not planning on doing automatic savestate loading for the initial setup.  
Taking
shots from the intro and demo should suffice, since the demos and the actual 
game use
the same engine and would be susceptible to the same bugs.  It is possible to 
add
automatic savestate loading at a later date to test specific problems if 
necessary,
but right now I'd rather spend the extra time working on other fixes and 
features.

Original comment by richard...@gmail.com on 19 Apr 2008 at 6:26

[GoogleCodeExporter]

Just certain bugs don't show up in the intro / demos. The biggest I can think 
off 
the top of my head is Link's 2D preview in the pause menu. 

It's okay if the first version of automated testing doesn't get these. Sadly 
after 
using mupen with a wider number of ROMs, I see there are some more major 
graphcs 
bugs to deal with first.

Original comment by sknau...@wesleyan.edu on 19 Apr 2008 at 7:39

[GoogleCodeExporter]

What would this be written in? C?

Original comment by sgorman07@gmail.com on 19 Apr 2008 at 10:41

[GoogleCodeExporter]

Since it's mostly going to involve text processing and shell calls, I'll write 
the
regression test controller in Python.

Original comment by richard...@gmail.com on 20 Apr 2008 at 12:22

[GoogleCodeExporter]

Heh. While I have no experience with emulation and almost none with C, I do 
like 
Mupen64Plus and I'm fairly good with Python. Maybe I'll make some time to help 
out 
during my summer vacation.

Original comment by stephan....@gmail.com on 22 Apr 2008 at 3:35

[GoogleCodeExporter]

My screenshot modification should help you out in this case, this will allow you
categorize the screenshots easier for review.

Original comment by sgorman07@gmail.com on 23 Apr 2008 at 8:34

[GoogleCodeExporter]

The nightly auto-regression is up and running as of April 28; the script code 
and
initial configuration file were committed in rev 324.  There are 658 
screenshots from
36 games taken and compared against reference images.  A test report including 
build
warnings/errors and any differences between test and reference images is 
emailed to
developers, and the images and report are archived on my local PC.

Original comment by richard...@gmail.com on 28 Apr 2008 at 2:14

• Changed state: Fixed