OpenAdaptAI / OpenAdapt

AI-First Process Automation with Large ([Language (LLMs) / Action (LAMs) / Multimodal (LMMs)] / Visual Language (VLMs)) Models
https://www.OpenAdapt.AI
MIT License
735 stars 95 forks source link

Implement PyAutoGuiReplayStrategy #770

Open abrichr opened 1 week ago

abrichr commented 1 week ago

Feature request

We would like to support prompting the model to generate PyAutoGUI code, like in OSWorld and elsewhere.

This involves:

Motivation

Compare performance of different action spaces.