Web Automation Agent MVP

[drobnikj]

The features of MVP for browser automation agent

Note here that we are not aiming for a production-ready tool. This is more of an experiment to test what browser automation using GPT powered by crawlee can handle.

Code related

• [x] It is better than https://github.com/unconv/puppeteer-gpt
• [x] It can perform actions based on the user's natural language instructions
• [ ] List of actions:
  • [x] Go to the URL
  • [x] click the link
  • [ ] It fills and submits the form - this is tricky for JS-like forms. We need to know how to fill out the form first and send it to LLM
  • [x] extract data from the page
  • [x] save data into the output
  • [x] save data into a dataset
  • [ ] perform google search
  • [x] action to take screenshot
  • [ ] action to download the file to kvs
• [x] It has a live view where the user can see what agent is browsing using the Apify live view
• [x] It can close cookies consent
• [ ] It shrinks HTML before it sends it to LLM to save the token
• [ ] It has secret input for password and can perform sign-in flows with username/password

No code

• [ ] Create a set of tests, around 20 cases on which we will test the Actor. The best way is if we manage to get some from Apify customers.
• [x] Figure out the name of the tool for the Apify store
• [x] Write readme

[mtrunkat]

@jirimoravcik @drobnikj

• I'd also teach him to save a screenshot or download a file to a key-value store, this may generate some nice use-cases
• I think that without external memory or some better shrinking of prior actions, we won't move forward, as the scope will stay very limited

[drobnikj]

@mtrunkat

• some better shrinking of prior actions

We solved it in https://github.com/apify/actor-web-automation-agent/pull/1/files for now. It removes HTML from previous action and keep just HTML for next action.

[mtrunkat]

Cool. We will eventually need to add memory anyway but we can leave that for later phases.

[drobnikj]

The Actor was published https://apify.com/apify/ai-web-agent

[drobnikj]

I will close this MVP task and move think which we didn't solve in MPV into separate task.