Support the VICE test bench?

dirkwhoffmann commented 1 week ago

Lately, there has been some discussion about whether the VICE testbench should be supported.

I've tried that in the past but gave up at some point, mainly because I am poor at reading shell scripts. The VICE testbench consists of script files, running the emulator with the VICE tests, and recording the outcome.

I have done some preliminary work in case someone wants to get involved and help out.

VirtualC64 can now be configured via the command line

To be as flexible as possible, arbitrary RetroShell commands can be passed in: Syntax: virtualc64 -"<RetroShell command 1>" -"<RetroShell command 2>" etc. VirtualC64 will collect and execute all provided arguments as a RetroShell script on startup.
The debugcart feature is supported

It is deactivated by default. To enable it, set OPT_EXP_DEBUGCART to true (e.g., via the command line or inside RetroShell).

Debugcart is sniffing at address 0xD7FF. When this address is written, the emulator exists with the provided value as the return code.

mrdudz commented 1 week ago

If you implement the required features, i can add support for virtualc64 in the testbench - however, i don't have access to a macOS machine right now, so i can not test anything (or run the testbench with virtualc64 for that matter). If you can do that, i am more than happy to include your results to the testbench page on the VICE wiki of course :)

So, besides the debug-register (that you already have implemented) some more things are needed:

an option to reset all settings to their defaults, this will always be used as the first option on the cmdline, to make sure the config is always predictable (and we don't get surprises because of some user settings)
an option that will make the emulator quit (with exit status 1) after X cycles. This is needed for the case when a test program crashes or hangs, so the testbench can continue after a timeout
an option that will save a screenshot with a given name when (before) the emulator exits (make sure the current frame is actually completely rendered before that happens)
not required, but convenient: an option to make the emulator run "headless" (without creating a window) or at least start "minimized", that way the testbench can run in the background and you can still use the computer (it runs for over an hour)

Besides that, of course various other things need to be configurable... most importantly mounting disks and carts must be possible, and running programs - but i guess this is already the case :) Also some kind of "warp mode" is very useful here (else it takes hours for the testbench to finish)

However, i have no idea what you mean by your "Retroshell" examples - please point me to a list of all valid options and give some examples of how the resulting commandline would look like.

I'll prepare some rudimentary things in the meantime, it should be fairly straightforward to add :)

It'd also help if you checked out the testbench and make it work with any of the other supported emulators on your system (VICE is probably the most straightforward thing to do). If you do that in a svn tree, you can then just pull my updates later - and even produce patches easily should you find a bug.

mrdudz commented 1 week ago

I prepared some things already, see https://sourceforge.net/p/vice-emu/code/45328/ ... the only things missing now is fixing the commandline options to match whatever is needed in the virtualc64-hooks.sh file

Oh and i forgot above - please also point me to a list of features so i can filter the tests accordingly (what cartridges are supported, what other extensions, what drives, what image formats, what c64 models, pal/ntsc etc)

I have NOT yet added the recipes that process the generated output and produce the report (html or wiki pages) - that will be easier to fix when you have actually generated results :)

dirkwhoffmann commented 1 week ago

However, i have no idea what you mean by your "Retroshell" examples - please point me to a list of all valid options and give some examples of how the resulting commandline would look like.

I know you have no Mac, but the easiest way to learn about all available RetroShell commands is to enter the shell, hit TAB twice, or enter the beginning of a command and then hit TAB twice. It will auto-complete what has been typed in and print some help like this:

However, VirtualC64 comes with a small app called Headless, which is used to check the integrity of the build. One of its options is to run a RetroShell test script that executes all available commands to see if syntax errors occur. The script gives you an impression of the syntax and all available options.

A typical RetroShell script looks like this:

# Regression testing script for VirtualC64
# Dirk W. Hoffmann, 2021

# Setup the test environment
regression setup PAL

# Launch the test
regression run "/tmp/cpu_decimal.prg"

# Exit with a screenshot
wait 3 seconds
screenshot save "cpu_decimal"

In the meantime, I'll take care of the watchdog feature (exiting the emulator after a specific number of frames)...

dirkwhoffmann commented 1 week ago

Update: I've added the required infrastructure...

regression setup PAL This already existing command initializes the emulator with default settings and turns on warp mode. I am using this command in the existing regression tests. Other options than PAL are:
regression set DEBUGCART true

Activates the VICE debugcart feature: Writing a value into 0xD7FF terminates the emulator. The written value is passed back as the return code.
regression set WATCHDOG <cylces>

After initiating this command, the emulator will terminate after the specified number of CPU cycles with error code 1.
screenshot save <filename>

Saves the latest stable texture in raw format. If the provided filename is relative, it will be prepended by a default path, which is /tmp (to not break the existing regression tests). However, the path can be changed via screenshot set path.

Note: The screenshot command saves the texture in raw format. To convert it, e.g., into TIFF, ImageMagick can help out:

magick -depth 8 -size 716x285 rgb:/tmp/texture.raw texture.tiff

This command has been copied from a vAmiga Makefile. Hence, the texture size needs to be adjusted accordingly. VirtualC64 supports saving a cutout instead of the entire texture. The cutout size can be specified via the command screenshot set cutout.

For Mac users: The RetroShell commands regression and screenshot are only shown in the online help in debug builds. In release builds, they are registered as hidden commands. They are still available, though.

That's it for today.

mrdudz commented 1 week ago

If the provided filename is relative, it will be prepended by a default path, which is /tmp (to not break the existing regression tests). However, the path can be changed via screenshot set path.

Please don't - that will break the testbench. Relative path should be just that - the testbench handles whatever is needed. Using a separate option for the path will only make it more complicated.

dirkwhoffmann commented 1 week ago

Relative path should be just that - the testbench handles whatever is needed.

OK, no problem. The current code looks like this:

    // Add the target directory if the path is relative
    if (!filename.is_absolute()) filename = screenshotPath / filename;

    // Add a filename extension if none is specified
    if (!filename.has_extension()) filename.replace_extension("raw");

Setting the path to an empty string via screenshot set path "" will cause relative paths to stay relative.

mrdudz commented 1 week ago

ok, fine... i will try to update that script a bit, then you can have a look and see what happens :)

mithrendal commented 1 week ago

I did the following

cloned the testprogs folder from svn
in folder testprogs/testbench I renamed the example config to Makefile.config
set VICEDIR to vice 3.8 installation folder

# should point to the directory contain VICE x64 binary.
# if you have checked out "trunk" and "testprogs" into the same directory AND
# built the VICE binaries in-tree, you may leave this empty to use these 
# binaries.
#VICEDIR="/usr/local/bin"
VICEDIR="/Users/c64/projects/vice/bin"

inside VICEDIR is the x64sc from vice3.8 mac sdl

when I directly call /Users/c64/projects/vice/bin/x64sc from shell, vice by itself just starts fine.

./testbench.sh x64sc

gives me this

./testbench.sh x64sc
running tests for x64sc (all):
make: svn: No such file or directory
make[1]: petcat: No such file or directory
make[1]: *** [format.prg] Error 1
make: svn: No such file or directory
make[1]: acme: No such file or directory
make[1]: *** [tamtest.crt] Error 1
make: svn: No such file or directory
make: svn: No such file or directory
make: svn: No such file or directory
make[1]: acme: No such file or directory
make[1]: *** [rpm1.prg] Error 1
./selftest/ c64-pass.prg - error 
./selftest/ c64-fail.prg - error 
./selftest/ c64-pass.prg - 
error: call to ../../trunk/vice/src/x64sc failed.

I expected it to look into VICEDIR and start the tests ... what am I missing ?

mrdudz commented 1 week ago

oh well... you'll probably have to install a couple tools - i have never used the testbench on a box that isn't a development machine (which has all those tools) to be honest, so i never really noticed :) (i should put whatever comes out of this into the readme later... :))

a) if you call the testbench script directly (not via the Makefile), then you will have to give it the path in EMUDIR. ie call it like this:

EMUDIR=/Users/c64/projects/vice/bin ./testbench.sh x64sc

else it will look for the VICE binary in ../../trunk/vice/src/ - which would be the case when checking out the VICE repo and the testbench into the same toplevel directory (this is is the common development setup)

b) the tools that come with VICE have to be in the PATH (petcat, cartconv, c1541). This should be the case if VICE is properly "installed" on your system (i have no idea about using the macOS app thing, i guess you can add the path to your local bashrc)

c) as you see in the error messages.... you will also need the "acme" crossassembler and "subversion". acme builds the selftests and some helper programs (those it should actually skip when testing VICE...oh well) and svn is used to revert some changes to d64s that are being written to when testing. Admittedly both isnt the smartest solution - but it worked so far (i'll be happy to merge any patches that provide the same functionality and remove the requirement). I have no idea where to find macOS packages for those, but subversion must exist somewhere - and acme is trivial to build from source (there is also a macOS binary, although its a somewhat older version: https://sourceforge.net/projects/acme-crossass/files/macOS/ )

edit: oh, and it needs a c-compiler to build the "cmpscreens" program

mrdudz commented 1 week ago

So, i have committed some updates, changed the needed commandline options to what i think is the equivalent option for virtualc64.

The way commandline options work (with spaces in them and no clear separation between option and argument) makes this a very annoying quoting mess - there will certainly be errors left

Also, from the description above, it seems the screenshot feature doesn't do what it should do. What it should do is save a screenshot when the emulator exits - no more no less. No extra wait should be required before that, the testprogram will trigger exit by writing to the debug register, and then the screenshot should happen.

Also, if the order of those options somehow matters (i hope not), then the script will certainly need more adjustments.

I have marked a few places where i couldnt figure out what the respective option is with #TODO - mostly mounting disk images and cartridges IIRC

However, i recommend testing it like this:

EMUDIR="/home/foo/bar" ./testbench.sh virtualc64 selftest --verbose

That way you can (hopefully) see what is going on.

I'd try to make the regular (no screenshots) tests work first, that is probably only a matter of fixing some of quoting here and there

Once that is OK adjust the magick command and the offsets, and those tests should start working as well - provided the screenshot is saved in a "known" palette.

mithrendal commented 1 week ago

 EMUDIR=/Users/c64/projects/vice/bin ./testbench.sh x64sc
running tests for x64sc (all):
make: svn: No such file or directory
make: *** ../drive/format/: No such file or directory.  Stop.
make: svn: No such file or directory
make: *** ../C64/carts/tamtest/: No such file or directory.  Stop.
make: svn: No such file or directory
make: svn: No such file or directory
make: svn: No such file or directory
make: *** ../drive/rpm/: No such file or directory.  Stop.
./selftest/ c64-pass.prg - ok 
./selftest/ c64-fail.prg - error 
./selftest/ c64-pass.prg - ok 
./selftest/ c64-fail.prg - error 
(cartridge:c64-pass.crt) ./selftest/  - ok 
(cartridge:c64-fail.crt) ./selftest/  - error 
(cartridge:c64-pass.crt) ./selftest/  - ok 
(cartridge:c64-fail.crt) ./selftest/  - error 
../C64/raminitpattern/ cyberloadtest.prg - timeout 
../C64/raminitpattern/ darkstarbbstest.prg - timeout 
../C64/raminitpattern/ pattern00ff.prg -  analyzer (skipped)
../C64/raminitpattern/ platoontest.prg - timeout 
../C64/raminitpattern/ typicaltest.prg - timeout 
../C64/autostart/basic/ basictest.prg - timeout 
../C64/autostart/basic/ printpoint.prg - timeout 
../C64/autostart/basic/ printpoint2.prg - timeout 
../C64/autostart/defaults/ test.prg - timeout 
(disk:test.d64) ../C64/autostart/defaults/ test.prg - timeout 
../C64/autostart/random/ raster.prg -  interactive (skipped)
../C64/bankio/ bankio.prg - timeout 
../C64/openio/ dadb.prg -  interactive (skipped)
../C64/openio/ de00all.prg -  interactive (skipped)
../C64/openio/ de00int.prg -  interactive (skipped)
../C64/openio/ trivial.prg -  interactive (skipped)
../C64/openio/ gauntlet.prg - timeout 
../crtemulation/paltest/ paltest.prg -  interactive (skipped)
../crtemulation/ emusuxX0r-crest.prg -  interactive (skipped)
../general/banking00/ banking00.prg - timeout 
../general/Lorenz-2.15/src/ start.prg - timeout 
../general/Lorenz-2.15/src/ ldab.prg - timeout 
../general/Lorenz-2.15/src/ ldaz.prg - timeout 
../general/Lorenz-2.15/src/ ldazx.prg - timeout 
../general/Lorenz-2.15/src/ ldaa.prg - timeout 
../general/Lorenz-2.15/src/ ldaax.prg - timeout 
../general/Lorenz-2.15/src/ ldaay.prg - timeout 
../general/Lorenz-2.15/src/ ldaix.prg - timeout 
...

how to stop the skript ?

mrdudz commented 1 week ago

CTRL-C in the terminal

Also if i understood correctly, the watchdog option should be last on the commandline? and perhaps the quoting must be fixed somehow so the timeout value is inside the quotes in the resulting commandline

mithrendal commented 1 week ago

timeout was a stupid error by me... it did not found the .prg files ... now I corrected the paths

looks already a bit better now ✌️

mrdudz commented 1 week ago

yup :) you should also fix that svn error (ie install svn cmdline utils) - else some tests will randomly break after first run of the testbench

dirkwhoffmann / virtualc64

Support the VICE test bench? #810