search

web-arena-x / webarena

Code repo for "WebArena: A Realistic Web Environment for Building Autonomous Agents"
https://webarena.dev
Apache License 2.0
647 stars 94 forks source link

Will the trace of human annotation process be released? #73

Closed Junpliu closed 7 months ago

Junpliu commented 7 months ago

Thanks for the great research!

The traces of models are pretty useful for me, and I wonder whether the trace of the human annotation process, i.e., 810 test configs, will be released.

shuyanzhou commented 7 months ago

Hi, we have ~170 trajectories available in the form of playwright traces. I will release them soon. Note that the recorded trajectories are pretty noisy and may require efforts to clean them up.

wookayin commented 6 months ago

Thanks for sharing the human trajectories https://drive.google.com/drive/folders/1NrN_sawtYK2V_uHnmmS8ugmGIKUAsPgt! But all the data here is given in a very raw, internal format, and not quite human-readable, so they are hardly useful. I tried to convert them into some useful data (i.e. like into action sequences or some "trajectory" data), but it's quite difficult to process the traces.

Nevertheless I find https://trace.playwright.dev/ and https://playwright.dev/docs/trace-viewer is helpful, as the data appears to be in the raw playwright trace format.

Do you have any advice here or would you be willing to provide some "processed" data that you might already have used as a part of experiments or evaluation?