Convert test harness into a series of unit tests

[1ec5]

Debug builds of AVIM have a built-in test harness that automatically types and verifies all the words in an input file (such as the FVDP corpus) in various configurations. The test harness made for an interesting demonstration years ago, but these days it makes much more sense to run unit tests on the command line, without Firefox running.

Before we can get conventional unit tests up and running, we should factor out the core input method engine #20 to remove the dependency on the DOM. The unit tests should be run via Spidermonkey or, failing that, some other JavaScript command line tool like node.

[1ec5]

• [x] Build test harness
• [x] Load FVDP tests
• [x] Devise tests for past bugs and regressions
• [x] Devise tests for bugs filed against other IMEs
• [ ] Remove old UI-based test harness
• [ ] Unit tests for ckSpell()
• [ ] Fuzzing?

[1ec5]

Travis now runs all the unit tests, including the entire FVDP corpus, on each push. These tests only exercise the transformer; we’re lacking coverage of the overlay and frame scripts. To test those, we’ll need some way of installing the extension into Firefox via the command line, and we’ll need some way to send keypress events to Firefox (headless testing?).

[kimkha]

I found Browserify can simulate a browser in command line: http://www.rexfeng.com/blog/2014/07/how-to-unit-test-your-js-and-use-it-in-the-browser/

Hope this help

[1ec5]

Thanks for the tip. Browserify is good for automated testing of a webpage in a browser environment. Travis already tests the only script that can be embedded in a webpage – transformer.js – in the exact same JavaScript environment as Firefox. What I'm mainly interested in is ensuring that the non-webpage parts – the extension itself – get testing.

Now that I've removed the old test harness from the bootstrapped branch, the outstanding task is to get Travis to run its built-in headless Firefox installation with a profile that includes AVIM. Perhaps node.js would be handy for processing a Web form that accepts input from this headless Firefox and relays it to the test harness.

The main benefit of such a setup would be that we would know immediately when Firefox removes a low-level API (#125) or a WYSIWYG editor library like Ace changes its API. Unfortunately, the most fragile part of AVIM historically has been support for Web apps like Google Docs and iCloud, where we rely on internal routines or IDs that could change without notice. These apps require accounts; I suspect it might not be kosher to set up an iCloud account essentially for automated testing of itself.