automated testing goals [by 2018-01-30]

[joncampbell123]

I've been bringing up automated testing a lot, I think it's time to start it.

Note that some tests will require the DOSBox-X integration device, which contains functions to support automated testing.

The testing programs will be written to test against DOSBox-X, as well as real hardware where possible, to validate emulation accuracy.

[joncampbell123]

DOSBox-X:

• Option at DOSBox-X command line (and in dosbox.conf) to ask all console output (through INT 21h) to be automatically mirrored to the log file where a testing script can see it.
• Built-in command on drive Z: to
  • Trigger screenshot (into capture folder)
  • Start/stop WAV, AVI, MTWAV, etc. recording
• Option at DOSBox-X command line to ask DOSBox-X to auto-terminate after a specified period of time (in case the test hangs). Time will be seconds of emulator time.

[joncampbell123]

Integration device:

• Watchdog timer
  • Client can ask watchdog timeout to trigger DOSBox-X to exit with an error code
  • Client can ask watchdog timeout to trigger emulator reset
  • Client can ask watchdog timeout to trigger non-maskable interrupt
• Guest read-only access to built-in DOSBox-X resources
  • 8x8, 8x14, and 8x16 VGA font bitmaps
• RENDER "framegrabber" to allow the guest code to grab frames from the SDL GUI's output so it can see the screen.
• Mixer recording interface to allow the guest code to hear mixer audio output so it can verify audio output is working as intended.
• Interface functions to read horizontal/vertical sync timing information from VGA emulation to allow validating video output.
• Debug interface to trigger state dump of DOSBox-X components to the logfile
  • State dump of PIC (requests pending, masked, in service, etc.)
• Debug interface to inject data into serial port emulation as if received by the UART, and to inject changes to incoming modem status lines.
• Debug interface to read emulator time, return value of PIC_FullIndex() converted to fixed point, so that automated testing can validate timings

[joncampbell123]

8253/8254 timer:

• Timer 0 countdown tests: run the timer through the entire range of intervals (0x10000 to 1) and watch it countdown to look for problems (Mode 2 and 3) [PC-98 and IBM PC]
• Timer 0 interrupt tests: run the timer through the entire range of intervals (0x10000 to 1) watch it count down, and check to see whether it fires the IRQ at each point that it hits zero (Mode 2 and 3) [PC-98 and IBM PC]
• Timer 0 one shot countdown tests [PC-98 and IBM PC]
• Timer 0 interval tests to verify timer clock is correct [DOSBox-X integration device only]
  • Test will validate correct IBM PC timer frequency, as well as correct 8MHz and 10MHz based timer frequencies on PC-98. The host script running the emulator for the test will also ask for WAV capture and verify the frequency.
• Timer 2 (IBM PC) or Timer 1 (PC-98) PC speaker test: Timer doesn't cycle if speaker off, and cycles when speaker on [PC-98 and IBM PC]
• Timer 2 (IBM PC) or Timer 1 (PC-98) function the same as Timer 0 given any countdown in Mode 2 and 3 and one-shot. [PC-98 and IBM PC]
• Timer tied to PC speaker generates 440Hz (A above middle C) tone when given a time constant known to generate that frequency. This version of the test program will use the DOSBox integration device to "hear" itself to verify the frequency. The script running the test will also use WAV capture to verify the frequency. [DOSBox integration device only]
• Timer 2 validate counting occurs when the counter clock gate is switched on even if the output gate to the speaker is switched off (counting but silent) [IBM PC only]
• Timer 2 validate no counting occurs when the counter clock gate is switched off, even if the output gate to the speaker is switched on (not counting, but would make sound if cycling) [IBM PC only]
• Timer 2 validate that both counter and output gates must be switched on to produce an audible beep. Use the integration device to "hear" the output in the test program. Use WAV capture during the test so the test script can hear it. [DOSBox integration device only, in IBM PC mode]
• "RealSound" digitized speech (Timer 2 PWM through PC speaker) test. Use integration device to "hear" the output to verify it. [DOSBox integration device only]

Manual testing:

• Timer tied to PC speaker generates 440Hz (A above middle C) tone when given a time constant known to generate that frequency. [PC-98 and IBM PC] Companion test program to run on the host will also generate this frequency through the sound card so a human can hear both and compare easily.
• Timer 2 validate that both counter and output gates must be switched on to produce an audible beep. The user must be given the option to switch the bits on and off and confirm that the beep is audible only if both are switched on. A logging function is to be provided to log the bits and the user's indication whether a beep was audible.
• "RealSound" digitized speech (Timer 2 PWM through PC speaker) test. Play a digital sample and have the user verify that it was audible and understandable. Provide adjustment keys for the user to indicate whether played too fast or too slow. [PC-98 and IBM PC]

Questions:

• Timer 2 on IBM PC: If you trick the timer into producing a TTL signal (5V) and gate off the counter so it stays that way, can you produce "speech" by rapidly toggling the output gate to the speaker? Do any DOS games do that? [IBM PC only]

[martinlindhe]

Not sure if useful, but I recently looked into making tests for a similar project.

Here's what I did:

Select a bunch of small software (i chose 256 byte demos from http://pouet.net/)

• Pick one, run for a deterministic amount, say 1M instructions.
• Render video frame as a image to disk.
• Calculate MD5 of the ram
• Compare with expected results

My demo collection can be found at https://github.com/martinlindhe/dos-software-decoding

I did my tests in the rust programming language, so that part might be useless here, but for completeness, this is the code I am referring to: https://github.com/martinlindhe/dustbox-rs/blob/master/src/gpu_test.rs#L76

[joncampbell123]

@martinlindhe The idea is to make tests specific to each part of the emulation. In the list above I make it clear to start with the PIT timer (8254) emulation.

[martinlindhe]

Indeed testing is useful on many levels, was just sharing some experience :-)