lavague-ai / LaVague

Large Action Model framework to develop AI Web Agents
https://docs.lavague.ai/en/latest/
Apache License 2.0
5.49k stars 505 forks source link

Handle multiple positions for click #592

Open adeprez opened 3 months ago

adeprez commented 3 months ago

Certain inline elements and partially visible elements become unclickable when their bounding box center is not accessible for click.

Case 1 : Inline elements:

When an element is inline (e.g., a paragraph that begins on one line and ends on the next line, with no text in the middle), the center of its bounding box might fall in an empty area. In such cases, clicking the element becomes problematic, as the center point is not a clickable area.

Case 2 : Partially visible elements:

When an element is partially visible (e.g. covered by something else), but one of its corners remains clickable.

To handle that, we should