action space of som can not solve the select element

web-arena-x / visualwebarena

VisualWebArena is a benchmark for multimodal agents.

https://jykoh.com/vwa

MIT License

232 stars 44 forks source link

action space of som can not solve the select element #54

Closed leoozy closed 2 months ago

leoozy commented 4 months ago

There is no index in the selections

leoozy commented 4 months ago

There is no idx for selections in 27

kohjingyu commented 4 months ago

This is a known issue with the current SoM markup, we're not sure there's a good way to fix it. It's possible for the model to select those things by typing into SELECT instead (or using the arrow keys).

boyugou commented 3 months ago

SELECT seemingly is not in current Action space. Is there any specific reason for this design? Sometimes Select Option is almost impossible to be finished by CLICK.