search

web-arena-x / webarena

Code repo for "WebArena: A Realistic Web Environment for Building Autonomous Agents"
https://webarena.dev
Apache License 2.0
647 stars 94 forks source link

Issue with Accessibility Tree Missing Information #70

Closed jingrsu closed 7 months ago

jingrsu commented 7 months ago

After executing the evaluation script using the following command:

python run.py \
  --instruction_path agent/prompts/jsons/p_cot_id_actree_2s.json \ # this is the reasoning agent prompt we used in the paper
  --test_start_idx 0 \
  --test_end_idx 100 \
  --model gpt-3.5-turbo \
  --result_dir output

I've observed discrepancies in the Accessibility Tree generated for certain HTML logs. Specifically, some elements present in the web page are missing in the Accessibility Tree.

Example Case:

An instance of this issue is evident in the output for render_23.html. The URL for this HTML log is: http://:7770/3-pack-samsung-galaxy-s6-screen-protector-nearpow-tempered-glass-screen-protector-with-9h-hardness-crystal-clear-easy-bubble-free-installation-scratch-resist.html In the html log render_23.html, as seen in the screenshot below, the content of T. Grannon's review is missing in the Accessibility Tree, though the title of this review is present. Additionally, the title of the subsequent review appears.

截屏2023-12-04 下午1 27 08

The actual result shows a missing entry for T. Grannon's review in the Accessibility Tree.

shuyanzhou commented 7 months ago

Thanks for the report, this is related to a bug in cleaning the AXTree.