search

aisingapore / TagUI

Free RPA tool by AI Singapore
Apache License 2.0
5.6k stars 580 forks source link

AI Recorder to record web and desktop workflows to various formats - to explore #1089

Closed kensoh closed 1 month ago

kensoh commented 2 years ago

recorder —> … bounding box, ml model

kensoh commented 2 years ago

Hi @ruthtxh I'm considering closing this for now since I don't have the bandwidth to add this now. If you are adding, the gist is creating a desktop recorder and automate creating the images for visual automation from user's clicks and keystrokes.

Ie need some image process logic (or ML of common UI elements as fallback) to know how far from the click point to crop the screen to use as the element image identifier. I believe it is technical possible to create a usable recorder.

The main downside of such an approach is ambiguity of recording, but users can always fine-tune by snapping on their own the incorrect generated snapshots. If done well, can work for both desktop and web applications.

Maybe I'll assign to you to review with PM if this is worth exploring.

kensoh commented 2 years ago

The various formats in the issue title mean exporting to TagUI human language .tag, .docx, .py (RPA for Python), .cs (TagUI for C#) etc. But the main benefit is for the human language users. The technical users might not require this recorder.

kensoh commented 1 month ago

Closing but please ping back if this is still relevant or an issue.