search

web-arena-x / webarena

Code repo for "WebArena: A Realistic Web Environment for Building Autonomous Agents"
https://webarena.dev
Apache License 2.0
676 stars 103 forks source link

task 615-619 labels have "locator" in format that raises error #53

Closed nicholaschenai closed 10 months ago

nicholaschenai commented 10 months ago

task 615-619 labels: all of them have

"locator": "[...document.querySelector('.submission__inner').querySelectorAll('[href],[src]')].map(elem => elem.getAttribute('href') || elem.getAttribute('src')).join(' ')",

however, in webarena/evaluation_harness/evaluators.py class HTMLContentExactEvaluator(Evaluator) line 236 on, locators are expected to be empty or startswith "document." or "func.". Hence, ValueError is raised for 615-619

shuyanzhou commented 10 months ago

Good catch! #54 has fixed it!