search

web-arena-x / webarena

Code repo for "WebArena: A Realistic Web Environment for Building Autonomous Agents"
https://webarena.dev
Apache License 2.0
676 stars 103 forks source link

Issue on gitlab evaluation #24

Closed ltzheng closed 11 months ago

ltzheng commented 1 year ago

When the task is to create a new project from the JEKYLL template in gitlab, the evaluation is based on the project's initial commit message "Initialized from 'JEKYLL' project template". However, this is different from the initial commit log for projects created using template Pages/Jekyll. This also happens for the HTML template.

Thanks!

shuyanzhou commented 1 year ago

Thanks! Great catch. I am creating an issue #25 to gather all such errors and we will try to fix all known issues as soon as possible. We will properly acknowledge any contributions to reporting/fixing such bugs in the repo's REAME file in the future.