ServiceNow / BrowserGym

BrowserGym, a gym environment for web task automation in the Chromium browser.
Other
269 stars 32 forks source link

Better Error messages when receiving timeouts #140

Open recursix opened 1 week ago

recursix commented 1 week ago

Timout errors is by far the #1 error message from browsergym. The agent would benefit from having more info about the cause of the timeout.

With some code on the browsergym side, we could possibly extract some useful information.

The timeout error message is also often long, perhaps it could be reduced in size.

recursix commented 4 days ago

Example of a not so useful timeout:

It tries to fill a textbox, but for some reason, it doesn't work. In this task you have to click on the box and wait for the menu to appear. Would there be a way to extract further properties of this element to give insights about what's going on?

image

AXTree: RootWebArea 'Book Flight Task', focused [17] heading 'Book Your One-Way Flight', visible [19] textbox 'From:' value='Latrobe, PA', clickable, visible StaticText 'Latrobe, PA' [21] textbox 'To:' value='Elmira, NY', clickable, visible, focused StaticText 'Elmira, NY' StaticText 'Departure Date' [25] textbox '', clickable, visible [27] button 'Search', clickable, visible [30] status '', live='assertive', atomic, relevant='additions' StaticText '1 result is available, use up and down arrow keys to navigate.' [31] list 'Elmira, NY (ELM)', clickable, visible [37] listitem '', visible ListMarker '' image '' StaticText 'Elmira, NY (ELM)' [32] status '', live='assertive', atomic, relevant='additions' StaticText '1 result is available, use up and down arrow keys to navigate.'