Set up unit tests for text manipulations

[kendonB]

@comodoro mentioned that he was going to think about how to set up unit tests for functions that modify or add text (which is the majority of the functionality of caster). Let's use this space to work out how to go about this.

• [ ] Get a working example done.
• [ ] Add a documentation page in the contributing section.

--- There is a **[$5 open bounty](https://www.bountysource.com/issues/75946405-set-up-unit-tests-for-text-manipulations?utm_campaign=plugin&utm_content=tracker%2F1825907&utm_medium=issues&utm_source=github)** on this issue. Add to the bounty at [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F1825907&utm_medium=issues&utm_source=github).

[kendonB]

https://www.bountysource.com/issues/75946405-set-up-unit-tests-for-text-manipulations

[comodoro]

Good idea to track it. I did not have anything fancy in mind, for starters just what already is in testutils.py+ runner Script ( not enumerating tests like the old one ) + tests in a new directory. Sorry I did not get to it yet.

[LexiconCode]

This seems relevant https://github.com/dictation-toolbox/dragonfly/issues/36 as a way to load and test grammars without an engine on Travis.

[kendonB]

from gitter:

I rather think Dragonfly's BringApp, since it should focus the window as well (a thought for bring me). I think Travis currently does not support GUI on Windows.

[kendonB]

from gitter:

Dane Finlay @Danesprite Jul 11 23:35 @kendonB If you want GUI testing like this, then it should probably be done locally, especially given there are bugs like #86. You could have a GUI window that captures events similar to xev for X11.

David Zurow @daanzu Jul 12 01:23 another option would be to have a set of mock Actions for testing that just write to an output buffer

Dane Finlay @Danesprite Jul 12 01:53 You could also mock the caster/dragonfly functions that send the input events, maybe with mock.patch()?

Jcaw @jcaw Jul 12 04:32 If you're willing to run the tests on Linux, you could have a look at the dogtail framework - it's a black-box Python gui test suite https://gitlab.com/dogtail/dogtail ripping some code out of their gedit example:

# Start gedit.
run('gedit')

# Set focus on gedit
focus.application('gedit')

# Focus gedit's text buffer.
focus.text()

# [test action here]

# [assert text in element matches target]

I don't know if a tool with exactly these features exists on Windows, at least in Python

Jcaw @jcaw Jul 12 04:40 It's a pretty funky way of testing but I'm not sure how easy it is to formalise action testing to begin with.

Jcaw @jcaw Jul 12 04:56 @kendonB You could test the final cursor position (and other aspects of the program's state) with dogtail.

LexiconCode @LexiconCode Jul 12 05:58 There are some advantages to a functional GUI testing suit such as Dogtail. Testing outputs and verifying assertions would be helpful. However it's quite limited text functionality for the full scope of Caster. Put simply spplication support combined with OSs are more likely to be a a greater source source of bugs long-term. The issue comes down to how applications combined with the OS interpret text input and keypresses. It could be anything ranging from the behavior of auto complete to verifying standard such as "c-z" Unfortunately behavior can even change between versions of those applications. I think leveraging both methods would be advantageous but GUI testing suit would be necessary. The downside of GUI testing suit would could be interpreted as the complexity of set up and the cost of either time or resources per test instance.

LexiconCode @LexiconCode Jul 12 06:21 You know I wonder if Sikuli, Repository could be leveraged for just our purposese being cross-platform in all although it's not a full test suit. Tests would follow along the pattern of executing a function verify final expected output with screenshot using sikulix. sikulix4python looks interesting but too much of a proof of concept probably to be usable ATM.

[LexiconCode]

@kendonB @comodoro For testing functions that manipulations text here is a great example TextFormat

[kendonB]

I was really envisaging something higher level than that that would take in a "spoken" phrase and check that the output matches what's expected.

A motivation was that there were some user contributions that didn't actually match the expected output.

When I was looking at this I was close to getting notepad.exe to pop up, then have the engine try the command, then save the file and check the output. It would be ideal to have a text editor simulation within python though

[LexiconCode]

Definitely needed and that's what I envision as well. Fortunately there's a little bit more info on possibly doing this cross-platform. The latest version of sikulix supports running RobotFramework text-scripts. I have to evaluate this further robotframework which utilizes Python looks very promising.