search

Significant-Gravitas / AutoGPT

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
https://agpt.co
MIT License
163.7k stars 43.48k forks source link

Web Navigation Challenge #3936

Closed waynehamadi closed 3 months ago

waynehamadi commented 1 year ago

Duplicates

Summary 💡

Auto-GPT should be able to navigate website, enter text and submit forms.

FYI @notskynet

Examples 🌈

No response

Motivation 🔦

No response

anonhostpi commented 1 year ago

I would really like to learn more about these challenges you keep coming up with. I'm assuming that these serve as guidelines for writing unit tests?

waynehamadi commented 1 year ago

yes exactly, but they're more end to end tests that unit tests. We test the system, we don't test the code. This way even if in 6 months we do a complete refactorisation the tests will still be valid, and they will allow us to refactor Auto-GPT. ping me on discord https://discord.gg/autogpt merwanehamadi

Boostrix commented 1 year ago

Auto-GPT should be able to navigate website, enter text and submit forms.

good thinking:

anonhostpi commented 1 year ago

It's probably best to use selenium awaiters for a portion of that, and prompt the human user to navigate those, as the human would be who is agreeing to them, not the bot.

Boostrix commented 1 year ago

those were based on observing the agent "browse" with the browser instance set to visible - it's not confirming anything, and also isn't using search facilities and also doesn't seem to be using sitemap.xml to find its way (personally, not a huge fan of using selenium for non-dynamic websites, where classic crawling should suffice)

waynehamadi commented 1 year ago

@anonhostpi @Boostrix I like that you think of how to solve this problem.

This issue is more about how to test whether the problem has been solved (aka create the challenge).

@NotSkynet is going to help us find good static website where we can test whether Auto-GPT is able to navigate.

Boostrix commented 1 year ago

just keep the browser open/visible to see for yourself that browsing could use some TLC ...

This issue is more about how to test whether the problem has been solved (aka create the challenge).

re navigation: randomly generate navbar structures using a nested loop with different labels and menus/sub menus, generate a matching sitemap.xml and use the two in combination to see if the agent is able to "visit" a certain part, each link would trigger the same python CGI script to tell the back-end which links were found/clicked. Evaluation-wise, we will then need to specify a goal "navigate to the contacts/team/about/company page", the crawler/selenium should be able to determine what link that is (we could use random file names here to ensure that the LLM isn't guessing). The Python CGI running inside HTTPServer would then register an even if the link was found or not. We could probably bootstrap the whole thing by copying a bunch of drupal/wordpress templates into a directory to have actual navigation bars that a python helper would then customize (randomize) with different tiles/descriptions and a href links

re the search form: probably analogous to the click form we talked about yesterday, just with the twist that it's a single SEARCH field (input text) and a simple python based back-end that we can execute via a conventional HTTPServer instance, at which point we can then hook up the whole thing to pytest, as before.

zachary-kaelan commented 1 year ago

We can just use the XPath //form | //input | //textarea | //button | //select | | //a[@href] with maybe a couple other things, to find all the important elements on the page it needs to know about for forms and navigation. Feed it the list of results and their paths, and it should be able to infer what it needs to do to navigate the webpage.

Boostrix commented 1 year ago

Yes, exactly what I suggested here: https://github.com/Significant-Gravitas/Auto-GPT/pull/3551#issuecomment-1537341661

I believe, this sort of feature could be useful in general, so we could just as well implement a more generic "browse_website" or extend it as needed (with support for xpath, like you say)

If this is augmented with sitemap.xml data, it's probably rather flexible as is:

browse_website <url> <focus> <constraints> [<xpath>, <use_sitemap.xml>]

zachary-kaelan commented 1 year ago

If this is augmented with sitemap.xml data, it's probably rather flexible as is:

browse_website <url> <focus> <constraints> [<xpath>, <use_sitemap.xml>]

Oh yeah, duh. I forgot sitemaps are a thing. We can integrate one of the many, many pre-existing crawling tools there are out there and automatically generate a sitemap for every site visited that doesn't already have one.

I was already thinking about overcomplicating things and indexing the pages on the fly ourselves when I remembered that search engines can do things like site:example.com and inurl:contacts.

Boostrix commented 1 year ago

there is one PR that maps HTTP Request to some custom commands for scraping purposes I believe: #2730 Also, this would seem like a really good idea: #2181

anonhostpi commented 1 year ago

For a collection of issues/PRs/discussions to base the challenge off of you could use my trackers: Gist (Alt)

You are more than welcome to use them as the "static website." recommend using the README.md file. I do update my tracker files frequently, but I imagine you could find a github xpath on it that doesn't change.

Boostrix commented 1 year ago

That gist is really looking good and super useful, must have been a ton of work, thanks for that !

(probably should be added to the wiki in a similar shape or form)

anonhostpi commented 1 year ago

That gist is really looking good and super useful, must have been a ton of work, thanks for that !

Thank you!

(probably should be added to the wiki in a similar shape or form)

I would have added it to the catalyzing and/or moderator page, but I think you have to have repo ownership privileges for that. You can't make a PR against a wiki, unfortunately.

anonhostpi commented 1 year ago

@Boostrix, if you are interested in contributing to my gist, you can take a look at https://github.com/anonhostpi/AUTOGPT.TRACKERS.

I use Github Copilot to autofill a lot of the data, and then use the powershell script https://github.com/anonhostpi/AUTOGPT.TRACKERS/blob/main/.SCRIPTS/CONTRIBUTE.ps1 to push updates to it.

waynehamadi commented 1 year ago

yeah great gist. Anyone up for the task on this challenge creation ? @NotSkynet did you find a website we can do that in ?

Boostrix commented 1 year ago

I would say, this could just as well be a static local website - i.e. via a Python HTTP Server that is locally running as part of the test suite, we talked about using that idea in the context of the "contact form".

So basically we need a pytest module that starts up a HTTP server to track "actions" (GET/POST requests).

To the agpt agent it should not be relevant if it's navigating google.com or 127.0.0.1 :-)

BaseInfinity commented 1 year ago

Oh I was able to get this working with ChromeGPT (https://github.com/richardyc/Chrome-GPT) using the AutoGPT agent.

I was able to login to Reddit with pretty basic sentence

Here's my demo: https://www.youtube.com/watch?v=RkpvyGla0PA

Not only that but I got extensions working so I was able to use the ScribeAI Plugin to create an auto generated guide (though basic)

Edit: I'm realizing this is talking about testing and not "is this possible", my bad

Boostrix commented 1 year ago

testing would be the level to see if any submissions solve the challenge or not - ideally, we would be able to throw different URLs at the challenge, with goals to navigate to some pre-defined page, and possibly "action" (contact form etc).

For starters/experiments, this could be based on localhost - but eventually, it will need to work using arbitrary websites. If you are working on this, you should inform @merwanehamadi so that he can update the list of challenges accordingly - to prevent others from working on the same problem, also do state clearly if you'd like to team up with others on the same challenge

waynehamadi commented 1 year ago

@Noots123 thanks for the suggestion ! Anyone knows what to do to create a challenge around this ? FYI @Boostrix @BaseInfinity @anonhostpi @zachary-kaelan

waynehamadi commented 1 year ago

Mind2Web is a ground breaking news for web navigation agents : https://osu-nlp-group.github.io/Mind2Web/ @xiang-deng, @ysu1989 thank you for your work, hopefully I will look into this soon

github-actions[bot] commented 10 months ago

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

BaseInfinity commented 9 months ago

I would love to get some momentum on this again. Recently I played around with HyperWriteAI and their autonomous browser atm is the best I have played with. Here's a small demo

HyperWriteAI is closed source and ChromeGPT is a bit slow so I think AutoGPT can make a splash here

I mentioned this to @Pwuts and he put it on his radar, he mentioned having a selenium wrapper that could do something similar to ChromeGPT's implementation: https://discord.com/channels/1092243196446249134/1111659953493651547/1149034954488037469

I don't see why AutoGPT can't have as good browser navigation as HyperWriteAI, and it would be a huge win for the OS community.

The whole reason why I've been going down this rabbit hole is to experiment with generating E2E tests given a "task". I've been able to prototype with ChromeGPT but would prefer a solution inside of the AutoGPT repo.

Plus I feel like one of the best things about autonomous agents is being able to browse the web effectively and is something people want which you can tell by the hype HyperwriteAI got when it released (it was number 4 on Product Hunt that day). General public went nuts over it understandably. Seeing your browser perform a task like a human in front of your eyes is powerful stuff lol

Any ways hopefully this sparks up some convo and momentum in the browser navigation area and if there's any ways I can help push it forward =)

github-actions[bot] commented 8 months ago

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

BaseInfinity commented 8 months ago

bump, unfortunately I haven't had time to dig into this

ballonJourn commented 6 months ago

Beautiful show ! That's my forte @BaseInfinity

BaseInfinity commented 6 months ago

@ballonJourn while I haven't been able to make progress in this repo, I have been able to make progress in others that support AutoGPT.

Here's my latest prototype, PlaywrightGPT: https://www.youtube.com/watch?v=DH9cIm1qfug

I imagine a future where tests can self repair themselves and even generate code diffs on code change in CI.

github-actions[bot] commented 4 months ago

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

github-actions[bot] commented 3 months ago

This issue was closed automatically because it has been stale for 10 days with no activity.