search

normal-computing / fuji-web

Fuji is an AI agent that lives in your browser's sidepanel. You can now get tasks done online with a single command!
Apache License 2.0
193 stars 13 forks source link

lookup interactive elements with CSS #110

Open mondaychen opened 2 months ago

mondaychen commented 2 months ago

Currently, we query interactive elements using HTML semantics. There are some websites that do not always follow the rules. For example, I've seen this kind of button

<div class="btn_primary inline-flex cursor-pointer items-center gap-1 text-center font-semibold">New Job</div> New Job

They are currently ignored by WebWand.

However, if we traverse the entire DOM, we can find these elements by checking whether the CSS style has cursor: pointer, and assume it is also interactive.

Depending on the website, this might get more things done, but this can also sometimes mark some unwanted things as interactive and confuse the agent.

We might make this an option in settings, or allow as part of per-website rule