Reproducibility of Webarena under high temperature

web-arena-x / webarena

Code repo for "WebArena: A Realistic Web Environment for Building Autonomous Agents"

https://webarena.dev

Apache License 2.0

676 stars 103 forks source link

Closed zwhe99 closed 9 months ago

zwhe99 commented 11 months ago

Thank you for your amazing work @shuyanzhou ! I notice that you set the temperature to 1.0. Will this make it difficult to reproduce the results?

foreverpiano commented 11 months ago

run for 10 times and choose the best as result XD

shuyanzhou commented 11 months ago

@zwhe99 This is a good point. We will report the number with temperature 0 on at least one model in the coming month.

shuyanzhou commented 9 months ago

As promised, we have added an experiment with t=0 and the results are in Table 5 in our updated version