search

nut-tree / nut.js

Native UI testing / controlling with node
https://nutjs.dev
2.13k stars 116 forks source link

Relative image match / click (like ui.vision's XClickRelative) #601

Open johtso opened 1 week ago

johtso commented 1 week ago

So, ui.vision has a really handy feature where you can take an image of a portion of the ui, draw a green box around some consistent anchor point, and a pink box around the area that you want to interact with.

An example is a label for a form field, and then clicking on the input box that contains changing / dynamic data.

Is this something that can currently be done with nut? I didn't see anything in the docs.

I don't know if ui.vision does anything more clever than matching on the area in the green box and then calculating an offset using the relative position of the pink box.

https://ui.vision/rpa/docs/xclick

image

s1hofmann commented 6 days ago

Hi @johtso 👋

I'll have to check out the link you provided in detail before I can answer this thoroughly. I'll take some time to do so, please be patient :)