Automate extraction and testing of BASIC programs for ROM/Core regressions

[gardners]

Describe where we can find the problematic topic We have LOTS of example BASIC programs in the manual. We should have an automatic means of testing them, for any change in output on different versions of ROMs / COREs.

Describe the solution you'd like A tool that can find and extract the BASIC programs (removing false-positives for assembly listings and the like), and then run them remotely on a real MEGA65, and do a digital screen-capture. Compare the screen captures with prior versions and report the differences.

Describe alternatives you've considered Do nothing, and require volunteers to periodically test them by hand.

Additional context N/A

[gardners]

Current state extracts program listings, and near perfectly classifies if they are BASIC, ASM or C. Only the BASIC programs are dumped into files for testing.

Testing will be done either by first tokenising the programs and then uploading them to a MEGA65, or perhaps more easily (if less efficient, in terms of test run time) via the remote typing interface. As the tests are intended to be run automatically, the slow testing speed should be acceptable in the first instance. Speeding up the testing will still be hhighly desirable, however. This will require a version of petcat or similar that supports all BASIC65 tokens.

[lgblgblgb]

@gardners I think, BASIC65 has a way to load untokenized text as BASIC program with the IMPORT command, or something like that, I always forgot the exact method. The other thing: if we want only test BASIC65/ROM functionality, maybe Xemu can be used some way (though Xemu cannot run easily "headless" because of the strong dependence on SDL, initialized renderers and output etc, but probably with some X virtual framebuffer, it can be tricked, as once I tried, IIRC). Sure, it's not a viable option for core testing though, but can be OK when the test is more about the functionality of the ROM, especially if the examples are simple to parse results, eg printouts on the screen (so a memdump at the end from the video memory can be compared to the desired result, and such things). Though - for sure - I can try to make Xemu a more "play as a testing tool" kind, as well, if it's needed, though I think currently it would be very hard to really run as "headless", but anyway, running on Xvfb or something is a viable option (even if it's kind of ugly solution). Some additional modifications are desired though, like the ability to exit from emulation dictated by the tested program at the end (or program controlled screenshot, memory dump, etc), and things like that, but that's not so hard to do.

UPDATE: it seems it can be done without Xvfb as well by using "dummy" SDL video driver. I think, it's even possible that I can provide a mode, when all speed control timing is eliminated thus running as maximum speed of emulation (currently it may cause some issues though in relation how RTC and TOD ticks, but otherwise should be fine). The question really if it's useful for anything that I should investigate more into this kind of usage of the emulator.

[nobruinfo]

@gardners @lgblgblgb Only slightly related and unfortunately running in Windows I've just uploaded a script to put subdirectories into a .D81 and filling them with a bunch of .BAS ASCII files, properly converted of course. Because at the moment Xemu doesn't have a run-and-close-itself option I simply use Windows' TASKKILL. Well, it's a quick solution and for me it serves my purpose. Take a look and have a giggle:

https://github.com/nobruinfo/xemu-beginner-s-doc