gasse
commented
2 months ago Hi @alzambranolu13 , it seems you are visiting a page with more than 26 iframes, which we did not expect would happen. Can you share the url your agent was visiting?
Closed alzambranolu13 closed 1 month ago
gasse
commented
2 months ago Hi @alzambranolu13 , it seems you are visiting a page with more than 26 iframes, which we did not expect would happen. Can you share the url your agent was visiting?
alzambranolu13
commented
2 months ago This is the url: https://m.independent.ie/life
alzambranolu13
commented
2 months ago Hello, Here are the complete logs of the whole interaction in case is useful:
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
INFO:browsergym.experiments.loop:Running experiment GenericAgentArgs_on_openended_769 in:
/home/alejandra/agentlab_results/ui_assistant_logs/2024-09-13_15-31-42_GenericAgentArgs_on_openended_769
WARNING:browsergym.core.env:Overriding the task's viewport parameter ({'width': 1280, 'height': 720} => {'width': 1500, 'height': 1280}). This might change the task's behaviour and difficulty.
WARNING:browsergym.core.env:Overriding the task's slow_mo parameter (1000 => 1000). This might change the task's behaviour and difficulty.
INFO:browsergym.core.chat:Waiting for message from user...
INFO:browsergym.core.chat:Message received.
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
INFO:browsergym.experiments.loop:I need to find the second news article in the Life Section of the Irish Independent. Since I am currently on a Google search page, I will first search for "Irish Independent Life Section" to locate the relevant page. After that, I can navigate to the Life Section and find the second article to summarize it.
action:
fill('96', 'Irish Independent Life Section')
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
INFO:browsergym.experiments.loop:I have filled in the search box with "Irish Independent Life Section" to find the relevant news articles. The next step is to click the "Google Search" button to execute the search and retrieve the articles from the Life Section of the Irish Independent.
action:
click('241')
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
INFO:browsergym.experiments.loop:I need to find the second news article in the Life Section of the Irish Independent. The current page shows various sections and links related to lifestyle news. I will look for a link that likely leads to the Life Section articles. The link labeled "Lifestyle News - Food, Home & Trends - Irish Independent" seems to be a good candidate to explore further for articles.
action:
click('568')
WARNING:browsergym.core.observation:Skipping frame '' for marking, seems problematic.
WARNING:browsergym.core.env:An error occured while extracting the dom and axtree. Retrying (4/5 tries left).
Error('Frame has been detached.')
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
INFO:browsergym.experiments.loop:I need to summarize the second news article in the Life Section of the Irish Independent. The second article is titled "Modern Morals: I gave my baby daughter such a unique name and then my colleague copied it – I’m in disbelief." I will click on this article to read its content and then summarize it.
action:
click('1523')
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
INFO:browsergym.experiments.loop:I need to summarize the second news article in the Life Section of the Irish Independent, which is titled "Modern Morals: I gave my baby daughter such a unique name and then my colleague copied it – I’m in disbelief." To do this, I will first click on the link to access the article and read its content.
action:
click('1523')
WARNING:browsergym.core.observation:Frame marking: not all elements have been visited by the intersection_observer after 1000 ms
WARNING:browsergym.core.observation:Frame marking: not all elements have been visited by the intersection_observer after 1000 ms
WARNING:browsergym.core.observation:Frame marking: not all elements have been visited by the intersection_observer after 1000 ms
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
INFO:browsergym.experiments.loop:I need to summarize the second news article in the Life Section of the Irish Independent. The second article is titled "Modern Morals: I gave my baby daughter such a unique name and then my colleague copied it – I’m in disbelief". I will click on this article link to access its content for summarization.
action:
click('1523')
WARNING:browsergym.core.observation:Extracted AXTree does not contain frameId '93C495D2C6A1CBAA2161AACA8B6EA6EF'
WARNING:browsergym.experiments.loop:Exception uncaught by agent or environment in task openended.
Error:
Frame has been detached.
Traceback (most recent call last):
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/browsergym/experiments/loop.py", line 215, in run
step_info.from_step(env, action, obs_preprocessor=agent.obs_preprocessor)
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/browsergym/experiments/loop.py", line 333, in from_step
self.obs, self.reward, self.terminated, self.truncated, env_info = env.step(action)
^^^^^^^^^^^^^^^^
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/gymnasium/wrappers/time_limit.py", line 57, in step
observation, reward, terminated, truncated, info = self.env.step(action)
^^^^^^^^^^^^^^^^^^^^^
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/gymnasium/wrappers/order_enforcing.py", line 56, in step
return self.env.step(action)
^^^^^^^^^^^^^^^^^^^^^
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/browsergym/core/env.py", line 383, in step
obs = self._get_obs()
^^^^^^^^^^^^^^^
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/browsergym/core/env.py", line 503, in _get_obs
_post_extract(self.page)
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/browsergym/core/observation.py", line 85, in _post_extract
sandbox_attr = frame.frame_element().get_attribute("sandbox")
^^^^^^^^^^^^^^^^^^^^^
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/playwright/sync_api/_generated.py", line 3635, in frame_element
return mapping.from_impl(self._sync(self._impl_obj.frame_element()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/playwright/_impl/_sync_base.py", line 109, in _sync
return task.result()
^^^^^^^^^^^^^
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/playwright/_impl/_frame.py", line 267, in frame_element
return from_channel(await self._channel.send("frameElement"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 61, in send
return await self._connection.wrap_api_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 490, in wrap_api_call
return await cb()
^^^^^^^^^^
File "/home/alejandra/Documents/AgentLab/agentlab/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 99, in inner_send
result = next(iter(done)).result()
^^^^^^^^^^^^^^^^^^^^^^^^^
playwright._impl._api_types.Error: Frame has been detached.Additional a video:
https://github.com/user-attachments/assets/91f4f7c5-7399-471d-a9a9-f87b8eea5fde
gasse
commented
1 month ago I just opened a PR that should solve these issues #143
@alzambranolu13 can you try on your side? Just checkout the branch gasse/patch_43 and then make install
gasse
commented
1 month ago @alzambranolu13 this should be solved by #143 , I'll close this issue. If you still encounter problems, please open a new one
When giving the following request: "Summarize the second news article in the Life section of the Irish Independent", the agent successfully opens correct URL and navigates to news article, after finding the information necessary the program shuts down and returns the following error.
I'm running the program on Ubuntu 24.04 with fresh install of agentlab with pip install -e .