web-arena-x / webarena

Code repo for "WebArena: A Realistic Web Environment for Building Autonomous Agents"
https://webarena.dev
Apache License 2.0
633 stars 90 forks source link

Sorting issue for task id 49 #135

Open afourney opened 2 months ago

afourney commented 2 months ago

The gold URL for task #49 contains a bug.

 {
    "sites": [
      "gitlab"
    ],
    "task_id": 45,
    "require_login": true,
    "storage_state": "./.auth/gitlab_state.json",
    "start_url": "__GITLAB__/a11yproject/a11yproject.com",
    "geolocation": null,
    "intent_template": "Check out the most recent open issues",
    "instantiation_dict": {},
    "intent": "Check out the most recent open issues",
    "require_reset": false,
    "eval": {
      "eval_types": [
        "url_match"
      ],
      "reference_answers": null,
      "reference_url": "__GITLAB__/a11yproject/a11yproject.com/-/issues/?sort=created_asc&state=opened",
      "program_html": [],
      "url_note": "GOLD in PRED"
    },
    "intent_template_id": 300
  }

Note that the question is about the most recent open issues. However the url specified "?sort=created_asc" ... which will sort by least-recent / oldest.

shuyanzhou commented 2 months ago

Good catch! Thank you!