Simpler method for finding NVDA objects

[ctoth]

Currently while developing addons, matching objects to perform operations on is quite ugly and manual, based on inspecting object attributes and searching for particular roles, ids, etc.

I propose a new approach based upon the Document Object Model, where NVDA Objects implement methods analogous to .matches and .querySelector on DOM objects.

Things to consider:

• Do CSS selectors provide enough syntax to adapt to NVDA objects to be the right approach?
• If yes, exactly which selector types map to which aspects of NVDA Objects?

Proposed syntax

Consider the code in miranda32.py appModule in chooseNVDAObjectOverlayClasses

I propose replacing this with something which looks like:

if obj.matches('.CListControl'):
  ...
elif obj.matches('.MButtonClass .TSButtonClass .CLCButtonClass'):
  ...
elif obj.matches('.ListBox #0'):
  ...

This should also work for .querySelector.

Let's brainstorm as to exactly what we would want to map our default CSS selector types like . and # to, then see if we can get a PR together.

[derekriemer]

I like this proposal. One thing which is going to be hard to work through here. Moving through a tree of objects in IAccessible or UIA requires at least one COM call per node, plus one more usually to fetch the info, unless that node is already in existence in NVDA. This may or may not be the case, because NVDA creates the parts of the tree it needs lazily. For example, obj.querySelector(".babyGrid") might require the following actions

1. Fetch children, at least one COM call.
2. is any of them a babygrid?
3. repeat querySelector on each child.

The problem also comes into play when there's recursion loops, @jcsteh helped me debug several of them in various addons. Basically, one object fetches a property, which then creates another object, where code from that objects chooseNVDAObjectOverlayClasses recursively causes other objects to be created. This is a problem when you have a call like obj.firstChild.next.firstChild.windowClassName, and later, obj.parent.next, but can creep up sometimes. Just something to be aware of.

For syntax, we have far more than UIA to match on. I have this in mind for now.

1. .blah for windowClassName.
2. ..blah, for UIAClassname

3. blah for UIA Control ID's.

4. blah for unique IA2 ID's. or, #blah, where blah is a number, and it's an iaccessible object.

5. [class=inputarea in ia2Attrs] for things that are non standard. That's a unique one, because ia2 has attributes in dictionary form, like a:b;c:d;

Other proposed ones:

6. <. or <#: my parent is query selector.
7. ->. ... <-. next or previous sibling is something.

[ctoth]

I like all of those, but could we abstract to a single kind of id? Or do we think that we would want to be able to match against a control with an iAccessible id and a uID id which are different is a useful thing to support? Same with classes really.

[LeonarddeR]

I also like this proposal. For query selectors, we could also copy the attribute matching from query selectors, which would give code like:

obj.querySelector("[name='someName'][role='button'][isFocusable]:not([hasFocus])")

I agree some things are problematic, especially where walking descendants of IAccessible objects gets into view. Many things can be optimized for that, though. For example, if we only want to have objects with a specific window class, we can just take the desktop objects windowHandle and recursively walk all the child windows without trying to create objects for them. This has almost no performance impact, especially not when compared to creating an IAccessible object for every single window.

[derekriemer]

@leonardder commented on Sep 25, 2017, 10:29 PM MDT:

I also like this proposal. For query selectors, we could also copy the attribute matching from query selectors, which would give code like:

obj.querySelector("[name='someName'][role='button'][isFocusable]:not([hasFocus])")

We'd need a reverse mapping from role name to role number (used internally) for this. Also, I'm not sure how you plan to do a general search like this quite yet without a huge perf hit, but if you can figure out a way to do it, greatness!

I agree some things are problematic, especially where walking descendants of IAccessible objects gets into view.

Also UIA gets ugly here.

Many things can be optimized for that, though. For example, if we only want to have objects with a specific window class, we can just take the desktop objects windowHandle and recursively walk all the child windows without trying to create objects for them. This has almost no performance impact, especially not when compared to creating an IAccessible object for every single window.

True. We can also resort to FindWindowX for this specific case. Also, UIA has some fancy api's for this, I.E. propertyConditions, and/or/not conditions.

[derekriemer]

@ctoth commented on Sep 25, 2017, 4:03 PM MDT:

I like all of those, but could we abstract to a single kind of id? Or do we think that we would want to be able to match against a control with an iAccessible id and a uID id which are different is a useful thing to support? Same with classes really.

I assume you mean UIA id? If so, probably, although there's UIA automation ID, and UIA Framework ID, so we'd be choosing automation ID, but you'd almost never match on framework ID.

[josephsl]

Hi,

Seven years later...

Is anyone still interested in the idea Chris proposed?

Thanks.