Testing: develop testing infrastructure to verify that all games boot

[tomcw]

I've been thinking a bit about testing (after raising the Tetris crash on boot bug #216)... how common is this kind of issue? And is it worth having some test infrastructure just to test that each game boots?

Thinking about this in terms of using an emulator (specifically AppleWin), then the following crude soak-test should be doable:

• have a generic save-state at the TR menu, with the key input buffer empty
• create a script (eg. python) that does:
• for-each of the 300+ games:
  • edit the yaml (text) save-state to input the game name + an ENTER key-waiting
  • launch AppleWin with the save-state, and new command line options to:
  • set a breakpoint when the game has loaded to its title screen
  • exit on breakpoint
  • exit on BRK (optional as some games may use this?)
  • exit after N cycles
  • save-state on exit
  • the game will boot (or not) and will exit on 1 of the above 3 conditions
  • script checks exit type == exit on breakpoint
  • script compares video memory (TEXT/HGR/DHGR) against a "ground truth" save-state to confirm that the game loaded successfully

This is admittedly pretty crude, and it doesn't check that the games are actually playable, but it gives a baseline of confidence that all the games at least boot.

NB. This is easily extensible to test different Apple II models (eg. Tetris has different HGR or DHGR title screen depending on Apple II model.

I can make the mods to AppleWin, if someone else wants to do all the script grunt-work(!)

[roughana]

I like the idea of automated testing, but in this scenario I’m not sure there would be much benefit. Once a game is in the system I don’t see changes happening to the system that would prevent 1/300 games from loading that this particular test proposal would catch. However, I expect that getting a game into the system would have lots of failure points that could/should be tested. Is there a PO? If so is there art? Is there a games conf entry? Prefs conf entry? Etc.

[frankmilliron]

Most already work. I'd prefer just testing each one, on each actual machine as AppleWin doesn't have //gs, //c, //c+ configs. Actually doesn't take that long, and it's what I'm already doing. I just haven't been spamming the bug reports as hard lately as I don't want to slow down porting. The majority of my bug list is just //c+ issues too, which is sort of a separate problem.

[tomcw]

Fair enough, and you guys have a better handle of where the test focus should be & where most issues are occurring.

I'm happy for someone to close this (eg. as "wontfix").

[roughana]

If we can trigger specific game starts via the launcher we could run build in an archive.org style emulator and capture screen shots of various screens (DHGR, Text2) that are problematic and compare with known good images. This would help confirm changes to VBL handling are ok.

[roughana]

There was a suggestion in Slack for a trigger to start a specific title. I suggested they create a feature request.