Mouse tracking does not work in Chromium

[snufas]

please look in to this problem, very important stuff,

Steps to reproduce:

this is a speech viewer log that should help to explain the problem.

Move navigator object to mouse Solutions for Visually Impaired text Move focus Solutions for Visually Impaired Move to focus Portable Video Magnification Systems - Ruby, Compact, and others link

what i am trying to get is read the object under the mouse

however, moving focus on it is not working. pressing NVDA+SHIFT+backspace does not focus on the object. other object types such as links, buttons are not working on any browser i tried. it works on desktop window for example without any problems. When I press nvda+shift+backspace it says the element name where it’s focused. i am trying to point object and refocus on the text as is showed on the speech log sample. when moving focus works it should tell where it focused however it just does not work in any browser. And focus stays where it was. NVDA version 2018.1 Windows version 10.0.16299 i have 2 improvements to suggest 1 clicking nvda+ctrl+right mouse click should automatically move the focus under mouse position. it would be very useful for people who has little vision left and can use the mouse pointing what they want to read. jaws have a very good function where pressing on the text moves the object and focus on it. 2 mouse following currently focused item there are some instances where using keyboard navigation in software NVDA does not read the item where as if mouse is used to hover under the program interface element, NVDA does read the item. by creating mode that mouse is moving with the currently focused item automatically would make things better in the situations where keyboard navigation does not produce any results. I have a little vision left, and there was and still is lots of situations where I must use mouse instead keyboard navigation because using arrows in nvda do not read the elements unless I hover the mouse under it. I will also provide some logs from log viewer with errors just not sure what action caused what error devs should be able to see what function in the software cause errors. i tested a lot of mouse focus events to see can i moove focus under cursor of the mouse.

logs ERROR - eventHandler.executeEvent (13:35:12.664): error executing event: mouseMove on <NVDAObjects.IAccessible.ia2Web.Ia2Web object at 0x073025B0> with extra args of {'y': 169, 'x': 249} Traceback (most recent call last): File "eventHandler.pyo", line 152, in executeEvent File "eventHandler.pyo", line 92, in init File "eventHandler.pyo", line 100, in next File "NVDAObjects__init.pyo", line 904, in event_mouseMove File "documentBase.pyo", line 24, in makeTextInfo File "textInfos\offsets.pyo", line 282, in init File "NVDAObjects\IAccessible__init__.pyo", line 113, in _getOffsetFromPoint COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None)) ERROR - eventHandler.executeEvent (13:43:56.497): error executing event: mouseMove on <NVDAObjects.IAccessible.ia2Web.Ia2Web object at 0x07237F70> with extra args of {'y': 241, 'x': 6} Traceback (most recent call last): File "eventHandler.pyo", line 152, in executeEvent File "eventHandler.pyo", line 92, in init File "eventHandler.pyo", line 100, in next File "NVDAObjects__init__.pyo", line 904, in event_mouseMove File "documentBase.pyo", line 24, in makeTextInfo File "textInfos\offsets.pyo", line 282, in init File "NVDAObjects\IAccessible__init__.pyo", line 113, in _getOffsetFromPoint COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None)) ERROR - eventHandler.executeEvent (13:43:57.006): error executing event: mouseMove on <NVDAObjects.IAccessible.ia2Web.Ia2Web object at 0x07237510> with extra args of {'y': 405, 'x': 348} Traceback (most recent call last): File "eventHandler.pyo", line 152, in executeEvent File "eventHandler.pyo", line 92, in init File "eventHandler.pyo", line 100, in next File "NVDAObjects__init__.pyo", line 904, in event_mouseMove File "documentBase.pyo", line 24, in makeTextInfo File "textInfos\offsets.pyo", line 282, in init File "NVDAObjects\IAccessible__init__.pyo", line 113, in _getOffsetFromPoint COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None)) ERROR - eventHandler.executeEvent (13:44:19.405): error executing event: mouseMove on <NVDAObjects.IAccessible.ia2Web.Ia2Web object at 0x072375B0> with extra args of {'y': 583, 'x': 590} Traceback (most recent call last): File "eventHandler.pyo", line 152, in executeEvent File "eventHandler.pyo", line 92, in init File "eventHandler.pyo", line 100, in next File "NVDAObjects__init__.pyo", line 904, in event_mouseMove File "documentBase.pyo", line 24, in makeTextInfo File "textInfos\offsets.pyo", line 282, in init File "NVDAObjects\IAccessible__init__.pyo", line 113, in _getOffsetFromPoint COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None)) ERROR - eventHandler.executeEvent (14:13:56.209): error executing event: mouseMove on <NVDAObjects.IAccessible.ia2Web.Ia2Web object at 0x07214F90> with extra args of {'y': 601, 'x': 575} Traceback (most recent call last): File "eventHandler.pyo", line 152, in executeEvent File "eventHandler.pyo", line 92, in init File "eventHandler.pyo", line 100, in next File "NVDAObjects__init__.pyo", line 904, in event_mouseMove File "documentBase.pyo", line 24, in makeTextInfo File "textInfos\offsets.pyo", line 282, in init File "NVDAObjects\IAccessible__init__.pyo", line 113, in _getOffsetFromPoint COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None)) ERROR - eventHandler.executeEvent (14:19:53.986): error executing event: mouseMove on <NVDAObjects.IAccessible.ia2Web.Ia2Web object at 0x07214550> with extra args of {'y': 300, 'x': 621} Traceback (most recent call last): File "eventHandler.pyo", line 152, in executeEvent File "eventHandler.pyo", line 92, in init File "eventHandler.pyo", line 100, in next File "NVDAObjects__init__.pyo", line 904, in event_mouseMove File "documentBase.pyo", line 24, in makeTextInfo File "textInfos\offsets.pyo", line 282, in init File "NVDAObjects\IAccessible__init__.pyo", line 113, in _getOffsetFromPoint COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None)) ERROR - eventHandler.executeEvent (14:25:57.253): error executing event: mouseMove on <NVDAObjects.IAccessible.ia2Web.Ia2Web object at 0x07214E30> with extra args of {'y': 389, 'x': 656} Traceback (most recent call last): File "eventHandler.pyo", line 152, in executeEvent File "eventHandler.pyo", line 92, in init File "eventHandler.pyo", line 100, in next File "NVDAObjects__init__.pyo", line 904, in event_mouseMove File "documentBase.pyo", line 24, in makeTextInfo File "textInfos\offsets.pyo", line 282, in init File "NVDAObjects\IAccessible\init__.pyo", line 113, in _getOffsetFromPoint COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None))

[Adriani90]

cc: @feerrenrut, @jcsteh this is quite related to #4896.

[Adriani90]

While in Firefox it seems to work quite reliable now, in other browsers it doesn't really works reliably. I guess this might be really hard to fix for all browsers.

[seanbudd]

I believe this needs an external fix from Chrome, once #13254 is merged. Firefox works as expected with #13254

[snufas]

@seanbudd Are you going to investigate this? This bug exists for good few years now. This affects all chromium-based browsers and apps. Used to work just fine before though, so I am not sure whether that's the fault of NVDA, or something changed in the chromium-based applications.

I would like to see the fix at some point, because I use mouse tracking quite often. "move navigator object to mouse" command works. But as you realize, it is definitely not the most comfortable way to use mouse tracking for sure.

[seanbudd]

@snufas - no plans to investigate this. as mentioned, I believe this needs an external fix in chromium

[snufas]

@seanbudd Well, I suppose, you have to investigate still, you cannot just give the bug to the chromeum team, and expect to find it themselves. I cannot track it down myself, but I am assuming you can, so please do the diligence, so it can be reported properly.

I would highly appreciate if someone more knowledgeable than I am, could track this down properly.I don't have have even the slightest clue what could I report to them.

[jcsteh]

1. The original steps in https://github.com/nvaccess/nvda/issues/8076#issue-304409271 don't work because you're trying to move focus to a piece of text. Text is not focusable in any browser. Focus refers to keyboard focus. This works on the desktop because desktop icons are focusable. The Move navigator object to mouse command should be sufficient to give you what you want. You can then use the review cursor to review the object under the mouse.
2. If you press move focus twice, that will move the browse mode cursor, which may be more what you want. The browse mode cursor can move anywhere, so whether something can get keyboard focus is not a concern.
3. That said, move focus twice may not work in Chromium in some cases because NVDA allows the mouse to hit text nodes in Chromium, but NVDA browse mode doesn't know how to handle text nodes. This part at least isn't a problem in Firefox because NVDA sets beTransparentToMouse to True for Mozilla text leaf nodes. It should probably do the same for Chromium.

[Adriani90]

@seanbudd could you explain why this needs an external fix? @jcsteh wrote:

3. That said, move focus twice may not work in Chromium in some cases because NVDA allows the mouse to hit text nodes in Chromium, but NVDA browse mode doesn't know how to handle text nodes. This part at least isn't a problem in Firefox because NVDA sets beTransparentToMouse to True for Mozilla text leaf nodes. It should probably do the same for Chromium.

So if this partially fixes the issue, then it sounds to me like it is an internal NVDA thing.

cc: @SaschaCowley

[Adriani90]

In general for better mouse handling in Chromium, I also created https://issues.chromium.org/issues/380416882 Feel free to add constructive input there, or complement what I've described in case there is something that I missed.