search

browserbase / stagehand

An AI web browsing framework focused on simplicity and extensibility.
https://www.browserbase.com
MIT License
1.17k stars 43 forks source link

Question: use vision AI with element position and click #149

Open guillegette opened 3 weeks ago

guillegette commented 3 weeks ago

Hi team,

Sorry for creating an issue about this, please let me know if there is a better place to share thoughts and questions.

I am really enjoying this project but I can already see many instances where the framework is not able to solve the request. Is there a reason why we couldn't use a AI vision model where we give a screenshot of the web page and ask "where is the red button", so we get coordinates back that we can then pass back to the browser and click on it?

Would love to hear your thoughts about this.

pkiv commented 3 weeks ago

Hey @guillegette! Thanks for sharing. I think this would be a great modifier to the useVision argument (useVisionCoordinates maybe?) We're discussing this in the community Slack here: https://stagehand-dev.slack.com/archives/C07UCP76U8G/p1730392286230649