search

ServiceNow / BrowserGym

BrowserGym, a gym environment for web task automation in the Chromium browser.
Other
327 stars 44 forks source link

Errors in visualwebarena #217

Closed recursix closed 4 days ago

recursix commented 3 weeks ago

16x : Exception uncaught by agent or environment in task .
TimeoutError:
Timeout 10000ms exceeded.
========================

Showing Max 2 stack traces:

2024-10-28 01:38:58,428 - 2180615 - browsergym.experiments.loop - INFO - Running experiment GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.282_22 in:
  /home/toolkit/agentlab_results/2024-10-24_21-33-14_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena/2024-10-28_01-38-10_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.282_22
2024-10-28 01:38:58,431 - 2180615 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:38:58,432 - 2180615 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:38:58,458 - 2180615 - browsergym.experiments.loop - DEBUG - Agent created.
2024-10-28 01:38:58,547 - 2180615 - browsergym.experiments.loop - DEBUG - Environment created.
2024-10-28 01:38:58,965 - 2180615 - asyncio - DEBUG - Using selector: EpollSelector
2024-10-28 01:39:01,707 - 2180615 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:39:01,708 - 2180615 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:39:01,717 - 2180615 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
...
...truncated middle of the log
...
2024-10-28 01:39:09,115 - 2180615 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/'>
2024-10-28 01:39:12,031 - 2180615 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/f/dataisbeautiful'>
2024-10-28 01:39:21,144 - 2180615 - browsergym.experiments.loop - WARNING - Exception uncaught by agent or environment in task visualwebarena.282.
TimeoutError:
Timeout 10000ms exceeded.
=========================== logs ===========================
navigating to "http://visualwebarena.eastus.cloudapp.azure.com:8080/f/dataisbeautiful", waiting until "load"
============================================================
Traceback (most recent call last):
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 216, in run
    step_info.from_reset(
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 405, in from_reset
    self.obs, env_info = env.reset(seed=seed)
                         ^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/time_limit.py", line 75, in reset
    return self.env.reset(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/order_enforcing.py", line 61, in reset
    return self.env.reset(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/core/src/browsergym/core/env.py", line 285, in reset
    task_goal, task_info = self.task.setup(page=self.page)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/visualwebarena/src/browsergym/visualwebarena/task.py", line 204, in setup
    page.goto(url)
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/sync_api/_generated.py", line 9303, in goto
    self._sync(
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_sync_base.py", line 109, in _sync
    return task.result()
           ^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_page.py", line 473, in goto
    return await self._main_frame.goto(**locals_to_params(locals()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_frame.py", line 138, in goto
    await self._channel.send("goto", locals_to_params(locals()))
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 61, in send
    return await self._connection.wrap_api_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 490, in wrap_api_call
    return await cb()
           ^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 99, in inner_send
    result = next(iter(done)).result()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
playwright._impl._api_types.TimeoutError: Timeout 10000ms exceeded.
=========================== logs ===========================
navigating to "http://visualwebarena.eastus.cloudapp.azure.com:8080/f/dataisbeautiful", waiting until "load"
============================================================

2024-10-28 01:38:58,366 - 2180618 - browsergym.experiments.loop - INFO - Running experiment GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.305_18 in:
  /home/toolkit/agentlab_results/2024-10-24_21-33-14_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena/2024-10-28_01-38-11_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.305_18
2024-10-28 01:38:58,368 - 2180618 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:38:58,370 - 2180618 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:38:58,394 - 2180618 - browsergym.experiments.loop - DEBUG - Agent created.
2024-10-28 01:38:58,455 - 2180618 - browsergym.experiments.loop - DEBUG - Environment created.
2024-10-28 01:38:58,879 - 2180618 - asyncio - DEBUG - Using selector: EpollSelector
2024-10-28 01:39:01,629 - 2180618 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:39:01,629 - 2180618 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:39:01,638 - 2180618 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
...
...truncated middle of the log
...
2024-10-28 01:39:08,417 - 2180618 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/'>
2024-10-28 01:39:09,799 - 2180618 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/f/memes/new'>
2024-10-28 01:39:20,434 - 2180618 - browsergym.experiments.loop - WARNING - Exception uncaught by agent or environment in task visualwebarena.305.
TimeoutError:
Timeout 10000ms exceeded.
=========================== logs ===========================
navigating to "http://visualwebarena.eastus.cloudapp.azure.com:8080/f/memes/new", waiting until "load"
============================================================
Traceback (most recent call last):
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 216, in run
    step_info.from_reset(
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 405, in from_reset
    self.obs, env_info = env.reset(seed=seed)
                         ^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/time_limit.py", line 75, in reset
    return self.env.reset(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/order_enforcing.py", line 61, in reset
    return self.env.reset(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/core/src/browsergym/core/env.py", line 285, in reset
    task_goal, task_info = self.task.setup(page=self.page)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/visualwebarena/src/browsergym/visualwebarena/task.py", line 204, in setup
    page.goto(url)
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/sync_api/_generated.py", line 9303, in goto
    self._sync(
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_sync_base.py", line 109, in _sync
    return task.result()
           ^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_page.py", line 473, in goto
    return await self._main_frame.goto(**locals_to_params(locals()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_frame.py", line 138, in goto
    await self._channel.send("goto", locals_to_params(locals()))
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 61, in send
    return await self._connection.wrap_api_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 490, in wrap_api_call
    return await cb()
           ^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 99, in inner_send
    result = next(iter(done)).result()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
playwright._impl._api_types.TimeoutError: Timeout 10000ms exceeded.
=========================== logs ===========================
navigating to "http://visualwebarena.eastus.cloudapp.azure.com:8080/f/memes/new", waiting until "load"
============================================================

8x : Exception uncaught by agent or environment in task .
TypeError:
'NoneType' object is not callabl

Showing Max 2 stack traces:

2024-10-28 01:40:47,810 - 2180614 - browsergym.experiments.loop - INFO - Running experiment GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.597_12 in:
  /home/toolkit/agentlab_results/2024-10-24_21-33-14_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena/2024-10-28_01-38-11_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.597_12
2024-10-28 01:40:47,811 - 2180614 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:40:47,813 - 2180614 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:40:47,835 - 2180614 - browsergym.experiments.loop - DEBUG - Agent created.
2024-10-28 01:40:47,836 - 2180614 - browsergym.experiments.loop - DEBUG - Environment created.
2024-10-28 01:40:49,014 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='about:blank'>
2024-10-28 01:40:49,615 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/customer/account/login/'>
2024-10-28 01:40:50,545 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/customer/account/login/'>
2024-10-28 01:40:50,549 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/customer/account/login/'>
...
...truncated middle of the log
...
action:
click('1625')  # This link appears to be for the Microsoft Xbox One Console with Accessories, 1TB HDD - Black.

2024-10-28 01:41:39,838 - 2180614 - browsergym.experiments.loop - DEBUG - Chat info sent.
2024-10-28 01:41:39,838 - 2180614 - browsergym.experiments.loop - DEBUG - Sending action to environment.
2024-10-28 01:41:39,838 - 2180614 - browsergym.core.env - DEBUG - Executing action
2024-10-28 01:41:39,943 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/catalogsearch/result/?q=xbox+one+console'>
2024-10-28 01:41:39,946 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/catalogsearch/result/?q=xbox+one+console'>
2024-10-28 01:41:39,948 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/catalogsearch/result/?q=xbox+one+console'>
2024-10-28 01:41:39,949 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/catalogsearch/result/?q=xbox+one+console'>
2024-10-28 01:41:39,951 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/catalogsearch/result/?q=xbox+one+console'>
2024-10-28 01:41:40,979 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/microsoft-xbox-one-console-with-accessories-1tb-hdd-black.html'>
2024-10-28 01:41:41,189 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/microsoft-xbox-one-console-with-accessories-1tb-hdd-black.html'>
2024-10-28 01:41:41,192 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/microsoft-xbox-one-console-with-accessories-1tb-hdd-black.html'>
2024-10-28 01:41:42,957 - 2180614 - browsergym.core.env - DEBUG - Action executed
2024-10-28 01:41:43,512 - 2180614 - browsergym.core.env - DEBUG - Active page checked
2024-10-28 01:41:43,513 - 2180614 - browsergym.core.env - DEBUG - User message done
2024-10-28 01:41:43,513 - 2180614 - browsergym.core.env - DEBUG - Initiating task validation
2024-10-28 01:41:43,691 - 2180614 - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): visualwebarena.eastus.cloudapp.azure.com:8082
2024-10-28 01:41:43,792 - 2180614 - urllib3.connectionpool - DEBUG - http://visualwebarena.eastus.cloudapp.azure.com:8082 "GET /media/catalog/product/cache/829a59e57f886f8cf0598ffca4f8a940/B/0/B00KL3WBBC.0.jpg HTTP/11" 200 14830
2024-10-28 01:41:43,860 - 2180614 - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): visualwebarena.eastus.cloudapp.azure.com:8082
2024-10-28 01:41:43,947 - 2180614 - urllib3.connectionpool - DEBUG - http://visualwebarena.eastus.cloudapp.azure.com:8082 "GET /media/catalog/product/cache/829a59e57f886f8cf0598ffca4f8a940/B/0/B00KL3WBBC.1.jpg HTTP/11" 200 17349
2024-10-28 01:41:43,976 - 2180614 - browsergym.experiments.loop - WARNING - Exception uncaught by agent or environment in task visualwebarena.597.
TypeError:
'NoneType' object is not callable
Traceback (most recent call last):
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 246, in run
    step_info.from_step(env, action, obs_preprocessor=agent.obs_preprocessor)
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 379, in from_step
    self.obs, self.reward, self.terminated, self.truncated, env_info = env.step(action)
                                                                       ^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/time_limit.py", line 57, in step
    observation, reward, terminated, truncated, info = self.env.step(action)
                                                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/order_enforcing.py", line 56, in step
    return self.env.step(action)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/core/src/browsergym/core/env.py", line 415, in step
    reward, done, user_message, task_info = self._task_validate()
                                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/core/src/browsergym/core/env.py", line 440, in _task_validate
    reward, done, user_message, info = self.task.validate(self.page, self.chat.messages)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/visualwebarena/src/browsergym/visualwebarena/task.py", line 259, in validate
    score = self.evaluator(
            ^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/visualwebarena/evaluation_harness/evaluators.py", line 636, in __call__
    cur_score = evaluator(trajectory, config_file, page)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<@beartype(visualwebarena.evaluation_harness.evaluators.PageImageEvaluator.__call__) at 0x7fb75721b2e0>", line 92, in __call__
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/visualwebarena/evaluation_harness/evaluators.py", line 576, in __call__
    pred_ans = self.captioning_fn(
               ^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable

2024-10-28 01:41:03,079 - 2186124 - browsergym.experiments.loop - INFO - Running experiment GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.598_10 in:
  /home/toolkit/agentlab_results/2024-10-24_21-33-14_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena/2024-10-28_01-38-11_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.598_10
2024-10-28 01:41:03,081 - 2186124 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:41:03,082 - 2186124 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:41:03,110 - 2186124 - browsergym.experiments.loop - DEBUG - Agent created.
2024-10-28 01:41:03,171 - 2186124 - browsergym.experiments.loop - DEBUG - Environment created.
2024-10-28 01:41:03,720 - 2186124 - asyncio - DEBUG - Using selector: EpollSelector
2024-10-28 01:41:07,604 - 2186124 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:41:07,605 - 2186124 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:41:07,615 - 2186124 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
...
...truncated middle of the log
...
action:
click('1528')  # Click on the first headphone option to view details.

2024-10-28 01:41:58,377 - 2186124 - browsergym.experiments.loop - DEBUG - Chat info sent.
2024-10-28 01:41:58,377 - 2186124 - browsergym.experiments.loop - DEBUG - Sending action to environment.
2024-10-28 01:41:58,377 - 2186124 - browsergym.core.env - DEBUG - Executing action
2024-10-28 01:41:58,494 - 2186124 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/catalogsearch/result/?q=Sony+headphones'>
2024-10-28 01:41:58,498 - 2186124 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/catalogsearch/result/?q=Sony+headphones'>
2024-10-28 01:41:58,499 - 2186124 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/catalogsearch/result/?q=Sony+headphones'>
2024-10-28 01:41:58,502 - 2186124 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/catalogsearch/result/?q=Sony+headphones'>
2024-10-28 01:41:58,503 - 2186124 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/catalogsearch/result/?q=Sony+headphones'>
2024-10-28 01:41:59,328 - 2186124 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/sony-zx110nc-noise-cancelling-headphones.html'>
2024-10-28 01:41:59,547 - 2186124 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/sony-zx110nc-noise-cancelling-headphones.html'>
2024-10-28 01:41:59,550 - 2186124 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8082/sony-zx110nc-noise-cancelling-headphones.html'>
2024-10-28 01:42:01,497 - 2186124 - browsergym.core.env - DEBUG - Action executed
2024-10-28 01:42:02,052 - 2186124 - browsergym.core.env - DEBUG - Active page checked
2024-10-28 01:42:02,052 - 2186124 - browsergym.core.env - DEBUG - User message done
2024-10-28 01:42:02,052 - 2186124 - browsergym.core.env - DEBUG - Initiating task validation
2024-10-28 01:42:02,222 - 2186124 - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): visualwebarena.eastus.cloudapp.azure.com:8082
2024-10-28 01:42:02,328 - 2186124 - urllib3.connectionpool - DEBUG - http://visualwebarena.eastus.cloudapp.azure.com:8082 "GET /media/catalog/product/cache/829a59e57f886f8cf0598ffca4f8a940/B/0/B01K5CTWPG.0.jpg HTTP/11" 200 17096
2024-10-28 01:42:02,417 - 2186124 - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): visualwebarena.eastus.cloudapp.azure.com:8082
2024-10-28 01:42:02,487 - 2186124 - urllib3.connectionpool - DEBUG - http://visualwebarena.eastus.cloudapp.azure.com:8082 "GET /media/catalog/product/cache/829a59e57f886f8cf0598ffca4f8a940/B/0/B01K5CTWPG.1.jpg HTTP/11" 200 18600
2024-10-28 01:42:02,518 - 2186124 - browsergym.experiments.loop - WARNING - Exception uncaught by agent or environment in task visualwebarena.598.
TypeError:
'NoneType' object is not callable
Traceback (most recent call last):
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 246, in run
    step_info.from_step(env, action, obs_preprocessor=agent.obs_preprocessor)
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 379, in from_step
    self.obs, self.reward, self.terminated, self.truncated, env_info = env.step(action)
                                                                       ^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/time_limit.py", line 57, in step
    observation, reward, terminated, truncated, info = self.env.step(action)
                                                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/order_enforcing.py", line 56, in step
    return self.env.step(action)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/core/src/browsergym/core/env.py", line 415, in step
    reward, done, user_message, task_info = self._task_validate()
                                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/core/src/browsergym/core/env.py", line 440, in _task_validate
    reward, done, user_message, info = self.task.validate(self.page, self.chat.messages)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/visualwebarena/src/browsergym/visualwebarena/task.py", line 259, in validate
    score = self.evaluator(
            ^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/visualwebarena/evaluation_harness/evaluators.py", line 636, in __call__
    cur_score = evaluator(trajectory, config_file, page)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<@beartype(visualwebarena.evaluation_harness.evaluators.PageImageEvaluator.__call__) at 0x7f2f38f731a0>", line 92, in __call__
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/visualwebarena/evaluation_harness/evaluators.py", line 576, in __call__
    pred_ans = self.captioning_fn(
               ^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable

2x : Exception uncaught by agent or environment in task .
Error:
Execution context was destroyed, most likely because of a navigatio

Showing Max 2 stack traces:

2024-10-28 01:38:58,353 - 2180614 - browsergym.experiments.loop - INFO - Running experiment GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.248_30 in:
  /home/toolkit/agentlab_results/2024-10-24_21-33-14_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena/2024-10-28_01-38-10_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.248_30
2024-10-28 01:38:58,355 - 2180614 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:38:58,356 - 2180614 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:38:58,381 - 2180614 - browsergym.experiments.loop - DEBUG - Agent created.
2024-10-28 01:38:58,440 - 2180614 - browsergym.experiments.loop - DEBUG - Environment created.
2024-10-28 01:38:58,841 - 2180614 - asyncio - DEBUG - Using selector: EpollSelector
2024-10-28 01:39:01,588 - 2180614 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:39:01,589 - 2180614 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:39:01,598 - 2180614 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
...
...truncated middle of the log
...
action:
click('324')

2024-10-28 01:39:35,218 - 2180614 - browsergym.experiments.loop - DEBUG - Chat info sent.
2024-10-28 01:39:35,218 - 2180614 - browsergym.experiments.loop - DEBUG - Sending action to environment.
2024-10-28 01:39:35,218 - 2180614 - browsergym.core.env - DEBUG - Executing action
2024-10-28 01:39:35,334 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums'>
2024-10-28 01:39:35,337 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums'>
2024-10-28 01:39:35,340 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums'>
2024-10-28 01:39:35,343 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums'>
2024-10-28 01:39:35,345 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums'>
2024-10-28 01:39:38,340 - 2180614 - browsergym.core.env - DEBUG - Action executed
2024-10-28 01:39:38,896 - 2180614 - browsergym.core.env - DEBUG - Active page checked
2024-10-28 01:39:38,896 - 2180614 - browsergym.core.env - DEBUG - User message done
2024-10-28 01:39:38,896 - 2180614 - browsergym.core.env - DEBUG - Initiating task validation
2024-10-28 01:39:41,532 - 2180614 - browsergym.experiments.loop - WARNING - Exception uncaught by agent or environment in task visualwebarena.248.
Error:
Execution context was destroyed, most likely because of a navigation
Traceback (most recent call last):
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 246, in run
    step_info.from_step(env, action, obs_preprocessor=agent.obs_preprocessor)
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 379, in from_step
    self.obs, self.reward, self.terminated, self.truncated, env_info = env.step(action)
                                                                       ^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/time_limit.py", line 57, in step
    observation, reward, terminated, truncated, info = self.env.step(action)
                                                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/order_enforcing.py", line 56, in step
    return self.env.step(action)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/core/src/browsergym/core/env.py", line 415, in step
    reward, done, user_message, task_info = self._task_validate()
                                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/core/src/browsergym/core/env.py", line 440, in _task_validate
    reward, done, user_message, info = self.task.validate(self.page, self.chat.messages)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/visualwebarena/src/browsergym/visualwebarena/task.py", line 259, in validate
    score = self.evaluator(
            ^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/visualwebarena/evaluation_harness/evaluators.py", line 636, in __call__
    cur_score = evaluator(trajectory, config_file, page)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<@beartype(visualwebarena.evaluation_harness.evaluators.PageImageEvaluator.__call__) at 0x7fb75721b2e0>", line 92, in __call__
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/visualwebarena/evaluation_harness/evaluators.py", line 532, in __call__
    elements = page.query_selector_all(locator)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/sync_api/_generated.py", line 8237, in query_selector_all
    self._sync(self._impl_obj.query_selector_all(selector=selector))
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_sync_base.py", line 109, in _sync
    return task.result()
           ^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_page.py", line 337, in query_selector_all
    return await self._main_frame.query_selector_all(selector)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_frame.py", line 304, in query_selector_all
    await self._channel.send("querySelectorAll", dict(selector=selector)),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 61, in send
    return await self._connection.wrap_api_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 490, in wrap_api_call
    return await cb()
           ^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 99, in inner_send
    result = next(iter(done)).result()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
playwright._impl._api_types.Error: Execution context was destroyed, most likely because of a navigation

2024-10-28 01:39:41,563 - 2180614 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/f/pics'>

2024-10-28 01:38:58,350 - 2180616 - browsergym.experiments.loop - INFO - Running experiment GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.249_32 in:
  /home/toolkit/agentlab_results/2024-10-24_21-33-14_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena/2024-10-28_01-38-10_GenericAgent-gpt-4o-mini-2024-07-18_on_visualwebarena.249_32
2024-10-28 01:38:58,351 - 2180616 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:38:58,352 - 2180616 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:38:58,375 - 2180616 - browsergym.experiments.loop - DEBUG - Agent created.
2024-10-28 01:38:58,434 - 2180616 - browsergym.experiments.loop - DEBUG - Environment created.
2024-10-28 01:38:58,841 - 2180616 - asyncio - DEBUG - Using selector: EpollSelector
2024-10-28 01:39:01,545 - 2180616 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
2024-10-28 01:39:01,546 - 2180616 - httpx - DEBUG - load_verify_locations cafile='/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/certifi/cacert.pem'
2024-10-28 01:39:01,555 - 2180616 - httpx - DEBUG - load_ssl_context verify=True cert=None trust_env=True http2=False
...
...truncated middle of the log
...
action:
click('110')

2024-10-28 01:39:17,341 - 2180616 - browsergym.experiments.loop - DEBUG - Chat info sent.
2024-10-28 01:39:17,341 - 2180616 - browsergym.experiments.loop - DEBUG - Sending action to environment.
2024-10-28 01:39:17,341 - 2180616 - browsergym.core.env - DEBUG - Executing action
2024-10-28 01:39:17,451 - 2180616 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums/all'>
2024-10-28 01:39:17,454 - 2180616 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums/all'>
2024-10-28 01:39:17,457 - 2180616 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums/all'>
2024-10-28 01:39:17,459 - 2180616 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums/all'>
2024-10-28 01:39:17,461 - 2180616 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums/all'>
2024-10-28 01:39:20,472 - 2180616 - browsergym.core.env - DEBUG - Action executed
2024-10-28 01:39:21,026 - 2180616 - browsergym.core.env - DEBUG - Active page checked
2024-10-28 01:39:21,026 - 2180616 - browsergym.core.env - DEBUG - User message done
2024-10-28 01:39:21,026 - 2180616 - browsergym.core.env - DEBUG - Initiating task validation
2024-10-28 01:39:21,149 - 2180616 - browsergym.experiments.loop - WARNING - Exception uncaught by agent or environment in task visualwebarena.249.
Error:
Execution context was destroyed, most likely because of a navigation
Traceback (most recent call last):
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 246, in run
    step_info.from_step(env, action, obs_preprocessor=agent.obs_preprocessor)
  File "/home/toolkit/dev/BrowserGym/browsergym/experiments/src/browsergym/experiments/loop.py", line 379, in from_step
    self.obs, self.reward, self.terminated, self.truncated, env_info = env.step(action)
                                                                       ^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/time_limit.py", line 57, in step
    observation, reward, terminated, truncated, info = self.env.step(action)
                                                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/gymnasium/wrappers/order_enforcing.py", line 56, in step
    return self.env.step(action)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/core/src/browsergym/core/env.py", line 415, in step
    reward, done, user_message, task_info = self._task_validate()
                                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/core/src/browsergym/core/env.py", line 440, in _task_validate
    reward, done, user_message, info = self.task.validate(self.page, self.chat.messages)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/dev/BrowserGym/browsergym/visualwebarena/src/browsergym/visualwebarena/task.py", line 259, in validate
    score = self.evaluator(
            ^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/visualwebarena/evaluation_harness/evaluators.py", line 636, in __call__
    cur_score = evaluator(trajectory, config_file, page)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<@beartype(visualwebarena.evaluation_harness.evaluators.PageImageEvaluator.__call__) at 0x7f52afb932e0>", line 92, in __call__
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/visualwebarena/evaluation_harness/evaluators.py", line 532, in __call__
    elements = page.query_selector_all(locator)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/sync_api/_generated.py", line 8237, in query_selector_all
    self._sync(self._impl_obj.query_selector_all(selector=selector))
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_sync_base.py", line 109, in _sync
    return task.result()
           ^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_page.py", line 337, in query_selector_all
    return await self._main_frame.query_selector_all(selector)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_frame.py", line 304, in query_selector_all
    await self._channel.send("querySelectorAll", dict(selector=selector)),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 61, in send
    return await self._connection.wrap_api_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 490, in wrap_api_call
    return await cb()
           ^^^^^^^^^^
  File "/home/toolkit/micromamba/envs/ui-assist/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 99, in inner_send
    result = next(iter(done)).result()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
playwright._impl._api_types.Error: Execution context was destroyed, most likely because of a navigation

2024-10-28 01:39:21,177 - 2180616 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums'>
2024-10-28 01:39:21,236 - 2180616 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums'>
2024-10-28 01:39:21,238 - 2180616 - browsergym.core.env - DEBUG - _activate_page_from_js(page) called, page=<Page url='http://visualwebarena.eastus.cloudapp.azure.com:8080/forums'>
gasse commented 4 days ago

254 #256 #257 #258 #259 should help reduce those errors.

I'll close this issue, please open a new one if you still have errors