Build test mode and test bed for major scripts

[phylll]

Testing even moderately complicated MMM scripts sucks, because the edge cases for specific rolls are impossible to force. Not testing at least moderately complicated MMM scripts sucks, too, because ... well, bugs. So, what to do?

It may just be possible to build a simple test suite by

1. inserting logic into each script that accepts special input from a set of special variables to take precedence over die rolls (or even user input)
2. build config scripts (one test script per actual script) to run through various test cases and overload those variables for testing.

How about Roll20 user input? If those Roll20 calls for input boxes or dropdowns remain in the code, Roll20 will execute one set of input controls before even calling MMM -- something I could live with if the values were properly overwritten by those supplied by the test environment...hmmm.

Roll20 can nicely simulate die rolls: [[20]] should create a critical success, for example. Not one that iscritical()/isfumble() would recognize, though. For those cases, we could use a loooong for loop to roll "real" dice until the right result pops up.

[phylll]

So overloading various kinds of inputs is possible using config scripts (one per test case), and minimal changes to every main script.

Die rolls may be an issue if we want to reliably test key edge cases, such as critical failures and successes. Maybe solvable somehow.

More importantly, however, I am making zero headway on how to check if the results of a script are actually those we expect, particularly in terms of output to the chat (I guess we could write little scripts to check if the changes made to my LP/AP are those we expect, so that's the smaller part of the problem). The relevant test cases and the number of scripts are already too many for us to just inspect the results and output manually -- the risk of something getting overlooked is as high as with manual testing. Yet there is no native way from within MMM to capture chat output and compare it to something (regexp).

One incomplete way out might be this: If we leave pretty optics and the testing of pure output issues aside (not an inconsiderable drawback), we could rework the scripts (not an inconsiderable effort) such that the key results (with these inputs, the script calculated these outputs) can be stored somewhere if "test mode" is on, e.g. in a character sheet. We could then build a test battery that runs all the test cases and then checks the storage sheet for the correct results. That would at least allow us to test for internal logic problems across a range of input conditions, including problems from sudden internal changes on Roll20's part.

But is this worth the effort? Looks like a LOT of work in terms of writing all the test cases manually, since there is no reusable code through functions etc., and building this additional functionality into the scripts, and STILL having to make sure manually that the output is correctly displayed every time we make a change that has anything to do with output.

[michael-buschbeck]

This is an intriguing train of thought.

I've been on and off thinking about native MMM features that would help testability, but (obviously) to no practical effect so far.

So. What do we want to test?

• Script logic, especially edge cases.

• Roll20 interaction (with under-the-hood API behavior and with state managed by Roll20, e.g. character sheets or token state).

What do we want to guard against?

• Home-baked script bugs
• MMM behavior changes and bugs
• Roll20 behavior changes and bugs

The good news is that we don't have to find a single approach that does all of these three things simultaneously.

For the first two (script bugs and MMM bugs) we could use a test harness of our own making. Notice how MMM's own test suite simply runs in any modern web browser of your choice – open test/scripts.html or test/expressions.html in your default browser and see. These tests run with the same MychsMacroMagic.js that runs in the Roll20 sandbox, against a (very crudely made and incomplete) mock-up of the Roll20 API (which, after all, is pure JavaScript as well).

For the third, we wouldn't have to test every edge case of every script at all. We know the touch points with Roll20 that can break our scripts: They're not that many, and it seems a small number of generalized test cases should be able to cover them (e.g. roll queries, nested roll queries, target selection, etc.). These test cases could probably be written as an MMM script that we'd just have to run once at the beginning of each game session to make sure Roll20 is still exactly as broken as we've come to expect it to be.

[phylll]

Well, MMM-native support would indeed make automated testing a LOT easier/more feasible :-)