search

OthersideAI / self-operating-computer

A framework to enable multimodal models to operate a computer.
https://www.hyperwriteai.com/self-operating-computer
MIT License
8.34k stars 1.11k forks source link

Google Search Results #99

Open TerminalGravity opened 7 months ago

TerminalGravity commented 7 months ago

When prompting 'operate' to search web, retrieve some info and save to {a google sheet} why not implement more crawling, parsing, retrieving and memory logic to improve the tooling for '[self-operating-computer]' ?

I prompted it with 'I want to research automated algorithmic trading strategies applying the VWAP standard deviation strategy then go to google sheets and save the information for short term trading strategies'

The automated system response performed the actions to open Chrome and search for a query; but I think it could perform alot more actions with more developed task funneling, parsing/retrieval/memory state management but when it came to searching google results or navigating to Google Sheets, it failed.

`❯ operate SYSTEM Darwin [Self-Operating Computer] Hello, I can help you with anything. What would you like done? [User] I want to research automated algorithmic trading strategies applying the VWAP st andard deviation strategy then go to google sheets and save the information for short term trading strategies [Self-Operating Computer] [Act] SEARCH Google Chrome [Self-Operating Computer] [Act] SEARCH COMPLETE Open program: Google Chrome [Self-Operating Computer] [Act] CLICK {'x': '50%', 'y': '9%', 'description': 'Click: Chrome Address Bar', 'reason': 'To type in the search query for researching VWAP standard deviation strategy'} [Self-Operating Computer] [Act] CLICK COMPLETE Click: Chrome Address Bar [Self-Operating Computer] [Act] TYPE VWAP standard deviation strategy for algorithmic trading [Self-Operating Computer] [Act] TYPE COMPLETE Type: VWAP standard deviation strategy for algorithmic trading [Self-Operating Computer] [Act] CLICK {'x': '17%', 'y': '28%', 'description': 'Click: First search result link', 'reason': 'To research automated algorithmic trading strategies applying the VWAP standard deviation strategy'} [Self-Operating Computer] [Act] CLICK COMPLETE Click: First search result link [Self-Operating Computer] [Act] CLICK {'x': '50%', 'y': '9%', 'description': 'Click: Chrome Address Bar', 'reason': 'To type in and navigate to Google Sheets'} [Self-Operating Computer] [Act] CLICK COMPLETE Click: Chrome Address Bar [Self-Operating Computer] [Act] TYPE https://sheets.google.com [Self-Operating Computer] [Act] TYPE COMPLETE Type: https://sheets.google.com [Self-Operating Computer] [Act] CLICK {'x': '50%', 'y': '9%', 'description': 'Click: Chrome Address Bar', 'reason': 'To enter the URL for Google Sheets'} [Self-Operating Computer] [Act] CLICK COMPLETE Click: Chrome Address Bar [Self-Operating Computer] [Act] TYPE https://sheets.google.com [Self-Operating Computer] [Act] TYPE COMPLETE Type: https://sheets.google.com ^C Exiting...

╭─  │  ~/R/self-operating-computer │   main !1  ╰─`

michaelhhogue commented 7 months ago

@TerminalGravity This is a good idea! If you did a PR on a memory/state management system, that would be awesome!