Extend UITester API

[kitchoi]

This issue laid out the few initial enhancements we need for the UITester proposed in #1107. (At the time of writing, the PR #1107 still needs a bit of refactoring / renaming of things.)

• [x] Define the methods in UITester and UIWrapper with basic implementation. (That is #1107)
• [x] Extend UITester to set a default registry containing implementations for TraitsUI editors.

Details

Since toolkit specific components may not be importable in an environment that does not have the toolkit, the content of the registry should be different depending on the toolkit being used. Which toolkit is used should be determined at runtime. At this step, the registry object can contain no or just a few implementations. The main goal here is to get the toolkit resolution setup.

• [x] Add the delay parameter to the UITester and allow it to be propagated to the subsequent UIWrapper.

Details

This feature allows developers to slow down the test run (down to every single keystroke or mouse click) as they write the test code so that they can verify the test code does what they want. In a typical usage, the delay will be removed after the developer is satisfied with their test code. The delay parameter should have a default value of no delay. We don't have to use this delay parameter at this step, the parameter will be used in the next step when we add actual implementations performing user interactions.

• [x] Implement MouseClick for ButtonEditor
  Details

This will be used for testing mouse clicking a button. This editor is chosen because it is very simple but also widely used.

One should then be able to write test code like this:

tester = UITester()
with tester.create_ui(obj) as ui:
    button = tester.find_by_name(ui, "some_name")
    with self.assertTraitsChanges(obj, "some_name", count=1):
        button.perform(command.MouseClick())

This should use the MouseClick command object being introduced in #1107 (at the time of writing, it is still there.)

It should be possible to slow down the test by setting the delay parameter in UITester (added in the previous step). And the test should be compatible with both Qt and Wx. Convert TraitsUI's own test for ButtonEditor.

• [x] Implement KeySequence, KeyClick and DisplayedText for TextEditor

Details

Similar to MouseClick, key clicking is a very basic operation for testing. By "key clicking", this means pressing and then immediately releasing a key. Note that "key clicking" is different from just "key press". Key press does not include the release, aka it could be holding a key down. In addition to the users causing changes via mouse clicks and key clicks, tests also need to inspect content being displayed on the GUI. This step also introduces a query: DisplayedText. Mouse click, key clicks are considered "commands". They introduce side-effects. Getting the displayed text causes no side-effect, and is considered a "query". Commands are expected to be called via `UIWrapper.perform`, and queries are expected to be executed via `UIWrapper.inspect`. Under the hood, they are implemented in exact the same way except one has returned value and the other does not. This distinction is entirely for code readability. At the end, we should be able to test TextEditor's 'auto_set' functionality with test code that looks like this: ``` foo = Foo(name="") view = View(Item(name="name", style="simple", editor=TextEditor(auto_set=False))) tester = UITester() with tester.create_ui(foo, dict(view=view)) as ui: tester.find_by_name(ui, "name").perform(KeySequence("NEW")) self.assertEqual(foo.name, "") tester.find_by_name(ui, "name").perform(KeyClick("Enter")) self.assertEqual(foo.name, "NEW") ``` And this: ``` foo = Foo(name="william") view = View(Item("name", format_func=lambda s: s.upper())) tester = UITester() with tester.create_ui(foo, dict(view=view)) as ui: display_name = tester.find_by_name(ui, "name").inspect(DisplayedText()) self.assertEqual(display_name, "WILLIAM") ``` Some of the tests in `traitsui.tests.editor.test_text_editor` and it should be possible to run the test under Qt or Wx.

• [x] Implement locating the textbox in RangeEditor and then perform KeySequence event on it.

Details

``` num = FloatWithRangeEditor() view = View(Item("number", editor=RangeEditor(low=0.0, high=12.0))) tester = UITester() with tester.create_ui(num) as ui: text = tester.find_by_name(ui, "number").locate(locator.WidgetType.textbox) text.perform(command.KeySequence("\b\b\b\b4")) text.perform(command.KeyClick("Enter")) self.assertEqual(num.number, 4) ``` This introduces a locator: locator.WidgetType We can worry about the slider part of this editor later. This step is for exercising the logic where we need to locate another widget within an editor. It is fairly typical for an editor to have multiple components. We should be able to convert a few of the existing tests within TraitsUI. Where there are existing bugs, we can either assert the buggy behaviour with a comment or skip the test with a comment.

• [x] Implement locating a nested UI for a custom style ListEditor. Consequently, one should be able to navigate into nested editors inside a nested UI.

Details

At the end, we should be able to write test code for a UI that uses custom ListEditor, like this: ``` tester = UITester() with tester.create_ui(obj) as ui: item = tester.find_by_name(ui, "list").locate(locator.Index(7)) item.find_by_name("name").perform(command.KeySequence("\b\b\b\b\b\bDavid")) self.assertEqual(obj.people[7].name, "David") ``` This will require adding another locator: locator.Index At the stage, testing the custom ListEditor does not support the locator.WidgetType introduced for RangeEditor. Likewise, testing RangeEditor does not support locator.Index. The following usage should raise with an error message that helps developers understand what locator objects are supported. e.g. Calling this with the CustomEditor should raise: ``` >>> tester.find_by_name(ui, "list").locate(locator.WidgetType.textbox) ... LocationNotSupported: is not supported for . Supported these: ``` Again one should be able to slow down the test for sanity checking using the delay parameter in UITester. But we probably don't need to delay the `locate` logic.

• [x] Implementing mouse click for CheckListEditor

Details

Note that the locating logic here will likely be slightly different from RangeEditor or custom ListEditor. For RangeEditor and custom ListEditor, we can throw away the locator.WidgetType or locator.Index information after calling `locate`. Here the information from the locator.Index needs to be retained for the subsequent mouse click. At the end, we should be able to write test code like this (the following is converted from an existing test): https://github.com/enthought/traitsui/blob/d73b8f14c1b181cadbdf27f629489ed032de60ff/traitsui/tests/editors/test_check_list_editor.py#L447-L457

• [x] Add "add_registry" method on UITester for extension.

Details

This allows the API to be extended rather than overridden. TraitsUI test code can also use this write test code that depends on less popular logic without necessarily exposing that into the public API (which will bring in backwards compatibility constraint). This should be a sanctioned way for downstream projects to extend the testing API. One should be able to define their own interaction, like this, while still make use of the implementations for other machinery: ``` def get_local_registry(): from traitsui.qt4.button_editor import SimpleEditor registry = InteractionRegistry() registry.register( editor_class=SimpleEditor, action_class=ManyMouseClick, handler=click_n_times, ) return registry class Test(TestCase): def test_my_button(self): tester = UITester() tester.add_registry(get_local_registry()) with tester.create_ui(app, dict(view=view)) as ui: tester.find_by_name(ui, "button").perform(ManyMouseClick(3)) self.assertEqual(order.submit_n_events, 3) ```

After completing the above steps, the most fundamental pieces of the testing API are available.

Final comments: We will need to continue adding more implementations for various editors. This can be done as part of the continuous effort in improving test coverage in TraitsUI, i.e. not within scope of this issue.

As these implementations grow, we can write integration tests for more complex application with intuitive and toolkit agnostic code. As an example, we will eventually be able test this demo in the future (note the above steps won't get us here just yet):

Show case for testing a demo example

https://github.com/enthought/traitsui/blob/d73b8f14c1b181cadbdf27f629489ed032de60ff/integrationtests/test_all_examples.py#L365-L413

[kitchoi]

Reminder: testing with wx might be somewhat platform dependent (testing with Qt is too, but less so). I have seen tests passing on Linux but failing miserably on OSX. While it would be nice to get wx test coverage as much as possible, in some circumstances, we might still need to exclude tests for certain toolkit and platform combinations to avoid being blocked.

[kitchoi]

Implement locating a nested UI for a custom style ListEditor. Here the information from the locator.Index needs to be retained for the key sequence / mouse click etc.

I was wrong. The custom ListEditor does not require the locator information to be retained. Editors like TableEditor and TreeEditor will require this locator retainment logic. I will update the description.

Edited: Done editing the description.

[rahulporuri]

@kitchoi is there a documentation step associated with each of the checkboxes/items or will documentation for the UITester be added separately, after all/most of the functionality has been implemented? Personally, it'd be better if we continuously add/update docs - which we can refine in the end.

[rahulporuri]

Also, how does this new test util work with ModalDialogTester/GuiTestAssistant from pyface? For projects which already use the pyface test utils, do we recommend porting their test code to use the new UITester?

[kitchoi]

@rahulporuri

is there a documentation step associated with each of the checkboxes/items or will documentation for the UITester be added separately, after all/most of the functionality has been implemented? Personally, it'd be better if we continuously add/update docs - which we can refine in the end.

I believe documentation is happening at the same time as the development is happening, if you count the documentation on individual objects and functions. UITester and the registry for extension have extensive docstring (most of the line changes in #1107 are documentation).

If you are thinking about the user manual, I am leaning towards having tested the UITester internally with TraitsUI tests first before advertising it more publicly, so we can still make changes without worrying too much about backward compatibility. Unlike other features in TraitsUI, the TraitsUI can be the users of its own testing tool.

I would propose having a separate issue for exposing UITester via an api module and adding description on the user manual. That would count towards a feature and therefore its timing might need to be evaluated separately.

In fact, we can add an empty api.py now to traitsui.testing package. Its emptiness would indicate what is not yet a published public API, whereas the absence of the module does not.

how does this new test util work with ModalDialogTester/GuiTestAssistant from pyface?

Yes.

For projects which already use the pyface test utils, do we recommend porting their test code to use the new UITester?

UITester does not replace ModalDialogTester nor GuiTestAssistant. So no porting is possible. The UITester here is also TraitsUI specific, whereas ModalDialogTester and GuiTestAssistant aren't. All three things can work together catering for different use cases.

[kitchoi]

I would propose having a separate issue for exposing UITester via an api module and adding description on the user manual. That would count towards a feature and therefore its timing might need to be evaluated separately. In fact, we can add an empty api.py now to traitsui.testing package. Its emptiness would indicate what is not yet a published public API, whereas the absence of the module does not.

@rahulporuri I forgot to add... :) What do you think about this plan?

[kitchoi]

how does this new test util work with ModalDialogTester/GuiTestAssistant from pyface?

Yes.

I overlooked the "How" bit in the question. I am sorry - my answer was so terrible.

Let me try again: Like ModalDialogTester can be used with GuiTestAssistant, UITester can be used with GuiTestAssistant. UITester does not depend on unittest.TestCase and can be used independently, e.g. pytest.

So I don't know what to say about "how", just use it when it suits your need? :)

[kitchoi]

@rahulporuri We talked about having an example that demonstrates testing various components of a more complex UI, sort of like an integration test. I opened https://github.com/enthought/traitsui/issues/1173 to collect tasks prior to exposing the API more publicly.

[rahulporuri]

sorry for completely missing this. the plan sounds good to me. looking at #1173