Terry

Terry is a virtual assistant that uses a composite robot learning approach to perform tasks on a computer using the same interface that a person would use. This interface is the operating system graphical user interface (OS GUI), including the peripherals needed to interact with it (the mouse and keyboard).

The composite learning approach attempts to incorporate aspects of three: demonstration, instruction, and reinforcement learning. The fact that Terry performs actions in the same way that a user would means those actions can be taught using demonstration learning. Additionally, a voice/text interface between Terry and the user will be used to both give commands and teach actions using instruction learning. Lastly, the user can watch the actions being carried out on-screen while Terry performs them, and provide feedback and guidance throughout.

CS Research Paper

The paper is in the docs/ folder, an exported PDF version of this Google doc online.

Demonstration Video

Here are demonstrations of Terry's major features implemented so far, done on a Mac laptop and a Windows desktop, uploaded to youtube

Credits

Icons

I got the following icons from fontawesome (link to their license here), which can be found in src/com/terry/res/img/:

• microphone (fa-microphone)

Color Palette

I used paletton to create the color scheme for Terry at docs/color_scheme.html.