Implement user guidance

[abrichr]

When the replay strategy determines that the current Screenshot/WindowState is sufficiently different from the expected state in the recording, we would like to prompt the user to take over and demonstrate how to complete the task in that situation:

1. compare the current state to the expected state in the current recording.
2. if not found, compare the current state to states in other recordings with the same description.
3. if not found, pause replay
4. notify the user that they need to demonstrate how to continue
5. start a new recording (with the same description as the one being recorded) as soon as the user starts generating InputEvents
6. wait for the user to finish the demonstration (e.g. by clicking on the system tray icon)
7. restart replay from where we left off

[Mustaballer]

Here are some questions and thoughts I had to address for some of these steps:

• We can compare the screenshot captured during the replay to the screenshot captured during the recording(and subsequent screenshots) using computer vision tool such as OpenCV. The issue is how can we determine if its sufficiently different.

  • Image comparison techniques such as thresholding and matching [1][2][3]. Thresholding involves setting a threshold value to determine if two images are similar or different, while matching involves comparing the features of two images using a distance calculation.
  • Is this what you had in mind?

• For the case when the user needs to demonstrate how to continue, we can use a desktop notification tool such as the WinToast library for Windows or the pynotify library for Linux(example below).

• For waiting for the user to finish the demonstration (e.g. 10s of inactivity), we can use a Python timer such as the time module.

[abrichr]

Also need to store this in the database, along with replays