OpenAdaptAI / OpenAdapt

Open Source Generative Process Automation (i.e. Generative RPA). AI-First Process Automation with Large ([Language (LLMs) / Action (LAMs) / Multimodal (LMMs)] / Visual Language (VLMs)) Models
https://www.OpenAdapt.AI
MIT License
960 stars 132 forks source link

Implement user guidance #104

Open abrichr opened 1 year ago

abrichr commented 1 year ago

When the replay strategy determines that the current Screenshot/WindowState is sufficiently different from the expected state in the recording, we would like to prompt the user to take over and demonstrate how to complete the task in that situation:

  1. compare the current state to the expected state in the current recording.
  2. if not found, compare the current state to states in other recordings with the same description.
  3. if not found, pause replay
  4. notify the user that they need to demonstrate how to continue
  5. start a new recording (with the same description as the one being recorded) as soon as the user starts generating InputEvents
  6. wait for the user to finish the demonstration (e.g. by clicking on the system tray icon)
  7. restart replay from where we left off
Mustaballer commented 1 year ago

Here are some questions and thoughts I had to address for some of these steps:

abrichr commented 1 year ago

Also need to store this in the database, along with replays