Prototype automated GUI tests

[mikej888]

dill:

we have testing for R components but nothing so far for Windows code. Do you know of a way in which we can test the Windows GUI? Len mentioned an “open source button clicker”?

[mikej888]

I checked http://en.wikipedia.org/wiki/List_of_GUI_testing_tools but the only open source automation or test tools were:

• AutoIt. https://www.autoitscript.com/site/autoit/. 2006-2014. Repetitive task automation. Closed source freeware.
• AutoHotKey. http://ahkscript.org/. 2006-2014. Open source fork of above. GPL. Scripts record applications to start, hotkeys to press, text to enter, coordinates of mouse clicks.
• Robot Framework. http://robotframework.org/. 2005-2014. Apache 2. Keyword driven acceptance test framework. For GUI testing, it uses AutoIt (https://code.google.com/p/robotframework-autoitlibrary/)

There is also:

• Sikuli. http://www.sikuli.org/. 2009-2013. MIT license. Uses image recognition. Kitware have a blog on using it within a test framework (http://www.kitware.com/blog/home/post/586).
• Qaliber, http://sourceforge.net/projects/qaliber. 2010-2013. GPL. Seems to have stagnated.

All other considerations aside, AutoHotKey is preferable to Sikuli as it seems less dependant upon the position and appearence of GUI elements than Sikuli.

As an intermediate phase to adopting automated GUI testing, documenting a series of manual protocols to work though, that cover the key functionalities of Distance, would be useful.

[mikej888]

Wrote scripts for Distance for Windows using AutoHotKey v1.1.16.0 - see autohotkey/ directory. There are three scripts: an example that starts and stops Distance, an example that creates a new project, and an example that implements Example 1 in the Distance for Windows 6.2 user's guide, plus a configuration and a utilities script.

To summarise my experiences:

Advantages:

• Very straightforward to get started with. A simple example that started and stopped Distance was written in 15 minutes or so from first downloading AutoHotKey.
• A rich set of commands for both invoking actions on a GUI (via keypresses or the mouse) and for other operations e.g. checking for the existence of files, deleting temporary files etc. See Command List for the current list.
• Support for modularity e.g. subroutines and include files.

Disadvantages:

• Very sensitive to window sizes if using relative positional commands for mouse moves and clicks. The scripts run fine on the Distance for Windows XP VM but not on my Windows 7 laptop as the widgets are a slightly different size. Positional commands for mouse moves and clicks can be avoided if there are keyboard alternatives available, but in Distance for Windows this was not always the case:
  • Selecting the Analyses tab in the Project Browser.
  • Selecting rows of the analysis table and double-clicking the gray ball to show the Analysis window.
  • Closing the Analysis window via clicking on its X icon.
  • The impact of these problems could be reduced by some reimplementation of Distance, to allow the above to all be done via keyboard shortcuts.
• Very sensitive to text in Windows. If windows don't have unique titles then determining which are active at any time must be done by the text that AutoHotKey can scrape from their content. Care must be taken to ensure that any text used to wait for windows to be come active, or to activate windows, is unique to that window (not just within the application being run but all the windows currently on the desktop). In such circumstances it's better for AutoHotKey to be run with a minimum of other applications open.
  • The impact of these problems could be reduced by some reimplementation of Distance, in terms of naming widgets.
• If there are no unique short-cuts for a button then there is a need, when writing a script, to count the number of TAB presses required to move between widgets so that a desired button, or other widget, is active. This can be very time consuming.
• Sometimes keyboard short-cuts can't be used because more than one widget shares the same short-cut e.g. in the Analysis window both Properties... buttons have the ALT+P short-cut.
  • This could be addressed by some reimplementation of Distance.
• The scripts are sensitive to changes in the GUI e.g. repositioning of widgets to be clicked by the mouse or changes in the TAB ordering of widgets.
• A lot of experimentation is required to ensure that enough time is given for the application to respond before invoking the next action. AutoHotKey supports a SetDelay command which can set a standard delay (in the scripts I've set this to 1000ms) but for certain activities (e.g. waiting for a run) extra time is needed. This requires some trial-and-error to identify suitable timings.

To summarise, I was impressed by how quickly one can write AutoHotKey scripts and they do support a rich set of commands. The key question for their adoption would be balancing the trade-off between:

• The time taken to write each script (implementing Example 1 took approximately 5 hours).
• The stability of the interface in terms of mouse positions, tab sequences and short-cut names.