OpenOffice 4.10: cannot get to next paragraph with braille scrollforward

[nvaccessAuto]

Reported by msuch on 2014-05-03 09:01 OpenOffice 4.10 seems to be perfectly accessible using NVDA. But here is the poblem I encounter: when scrolling thru a document with several paragraphs using the scrollforward key of a braille display, NVDA cannot get to the next 40 character at the end of paragraphs, as if it was the end of the document. A way to avoid this problem, si to first press ctrl+end to go to the end of the document and then ctrl+home to get to the top. Once you've done this, scolforward can go thru paragraphs as if you needed to load all the document.

[nvaccessAuto]

Comment 1 by mdcurran on 2015-01-26 06:19 technical: Open Office / Libre Office only create accessible objects for what is showing / was showing on the screen. next/flowsTo does not work after this point.

[Adriani90]

@msuch, @qchristensen is this issue still reproducible?

[msuch]

@Adriani90 It's been super long, I won't be able to reproduce this bug anymore.

[Qchristensen]

I haven't currently got a Braille display handy, but this appears to be caused by the same issue as: #4119, and #8673, so I suspect it probably is still present. Unless are you saying @msuch that you have tried nad can't reproduce this?

FWIS, I can still repro #4119 in Open Office 4.1.6 and LibreOffice 6.1.3.2 with NVDA 2018.4.1

[SaschaCowley]

@michaelweghorn I believe this issue is caused by 91739 – nvda can not read documents from top to bottom. The issue seems to be that LibreOffice sometimes doesn't correctly expose the IAccessible::FlowsFrom and IAccessible::flowsTo relations. Do you have any idea when this might be fixed?

In the meantime, I have a proof of concept workaround in the i4115 branch, but its implementation is complicated by having to walk the IAccessible tree to find the next/previous IAccessible leaf node. Any thoughts on whether this is a reasonable fix in the meantime?

I believe issues #4119 and #8673 have the same root cause.

Here is the primary snippet of concern for the WIP temporary fix. https://github.com/nvaccess/nvda/blob/5d9cea0100a1b2a0c4880b42e09fc0d18086537c/source/appModules/soffice.py#L238-L250

[michaelweghorn]

@michaelweghorn I believe this issue is caused by 91739 – nvda can not read documents from top to bottom. The issue seems to be that LibreOffice sometimes doesn't correctly expose the IAccessible::FlowsFrom and IAccessible::flowsTo relations. Do you have any idea when this might be fixed?

Support for browse mode/Some way of exposing the whole document including off-screen content in LibreOffice Writer is a topic that has received some attention recently and the Orca maintainer has also contacted me about it and there are some thoughts around it, but no ultimate way forward or timeline yet.

On of the concerns raised is performance. A related discussion on the LibreOffice development mailing list can be found in the thread starting here (which initially starts with Calc performance issues being raised): https://lists.freedesktop.org/archives/libreoffice/2024-June/092039.html

If you or somebody would like to get involved from the NVDA side, that would be great.

In the meantime, I have a proof of concept workaround in the i4115 branch, but its implementation is complicated by having to walk the IAccessible tree to find the next/previous IAccessible leaf node. Any thoughts on whether this is a reasonable fix in the meantime?

Ensuring the relevant content is/becomes visible on screen as long as off-screen content isn't part of the a11y tree sounds like a very good approach to me in general.

I don't have a braille display to test the particular issue here. In a quick test of that branch with nvda4115_sample_doc.odt, the "read from top to bottom" use case from https://bugs.documentfoundation.org/show_bug.cgi?id=91739 still doesn't seem to work fully, speech still stops somewhere towards the end of the first page for me.

[SaschaCowley]

@michaelweghorn upon further investigation, this is because we don't have a programmatic way (that I am aware of) to scroll so that the next page is in view. Are you aware of any such interface? Adding pages to the A11y tree as you suggest in the linked thread would be an option.

[michaelweghorn]

@michaelweghorn upon further investigation, this is because we don't have a programmatic way (that I am aware of) to scroll so that the next page is in view. Are you aware of any such interface?

@SaschaCowley No, unfortunately I'm not aware of anything. The only thing that comes to mind is IAccessibleText::scrollSubstringTo, but that again would require having the object (the next paragraph) so it can be scrolled into view, which is exactly what we're lacking here.

Maybe a workaround could be to send a key event to move the cursor down?

Interestingly, reading the whole document with Orca on Linux (with LibreOffice's gtk3 UI variant) works. From a first glance, what it seems to be doing is also using the flowsTo relation to get to the next object, setting the caret in that object, then retrieve the next object via the flowsTo relation,...

When testing with Accerciser, I can indeed navigate through the whole document by jumping from one object to the next following the flowsTo relation chain.

In NVDA's Python console, however, trying to get the <currentObject>.flowsTo doesn't always give the next paragraph, but returns None from some point on. That might be worth looking into on the LibreOffice side. (Why does it seem to behave differently on Linux than on Windows?)

[michaelweghorn]

When testing with Accerciser, I can indeed navigate through the whole document by jumping from one object to the next following the flowsTo relation chain.

In NVDA's Python console, however, trying to get the <currentObject>.flowsTo doesn't always give the next paragraph, but returns None from some point on. That might be worth looking into on the LibreOffice side. (Why does it seem to behave differently on Linux than on Windows?)

I've taken a closer look into that. As described before, retrieving the next paragraph like this returns None:

>>> focus.IAccessibleTextObject.text(0, -1)
'A way to avoid this problem, si to first press ctrl+end to go to the end of the document and then ctrl+home to get to the top.'
>>> focus.flowsTo

Using the low-level IAccessible2 methods directly shows that there is a flowsTo relation however:

>>> focus.IAccessibleObject
<POINTER(IAccessible2) ptr=0x9355d0c at 1037df0>
>>> focus.IAccessibleObject.nRelations
2
>>> focus.IAccessibleObject.relation(0)
<POINTER(IAccessibleRelation) ptr=0x943f464 at 1308760>
>>> rel = focus.IAccessibleObject.relation(0)
>>> rel.relationType
'flowsFrom'
>>> focus.IAccessibleObject.relation(1).relationType
'flowsTo'

From what I can see, the issue is that trying to retrieve the target fails in IAccessible._getIA2RelationFirstTarget: https://github.com/nvaccess/nvda/blob/df6bd0770db12ff73905251054d12f01911030dd/source/NVDAObjects/IAccessible/__init__.py#L1833-L1875

Both ways trying to retrieve the target IAccessible2 object fail. The first one, because the IAccessible2_2 interface is not implemented, and the second one because the call to target = relation.target(0) (in line 1866) triggers an error/exception.

I plan to take a look whether I can fix that on the LO side.

[michaelweghorn]

Both ways trying to retrieve the target IAccessible2 object fail. The first one, because the IAccessible2_2 interface is not implemented, and the second one because the call to target = relation.target(0) (in line 1866) triggers an error/exception.

I plan to take a look whether I can fix that on the LO side.

Pending LibreOffice change to fix the second case: https://gerrit.libreoffice.org/c/core/+/172215

With that in place, reading sample doc https://github.com/user-attachments/files/16609411/nvda4115_sample_doc.odt and the other one from LO bug https://bugs.documentfoundation.org/show_bug.cgi?id=91739 works as expected for me.

Could somebody please test the braille case once the fix is available in a current development build of LibreOffice? LO wiki entry explaining how to use daily builds for testing: https://wiki.documentfoundation.org/QA/Testing_Daily_Builds

[michaelweghorn]

Both ways trying to retrieve the target IAccessible2 object fail. The first one, because the IAccessible2_2 interface is not implemented, (...).

The pending change series up to https://gerrit.libreoffice.org/c/core/+/172258 now also implements IAccessible2_2::get_relationTargetsOfType, NVDA's preferred way to retrieve the next paragraph via the flowsTo relationship, for LibreOffice, so this also works without the fallback code path in NVDA (s. commit msg for details).

[michaelweghorn]

The LibreOffice changes have been merged now, so testing with a daily build from tomorrow or later would be appreciated. (I've explicitly tested the #4119 scenario and that works for me, and I very much expect that the root cause is the same, but don't have a braille device to explicitly retest this issue here.)

[SaschaCowley]

Thanks, @michaelweghorn, this fix seems to work for me. The only bit of oddness is when moving backwards in braille, when entering the previous page, NVDA shows the the name of the edit control and then the window, before then moving to the last line of the previous page. This only seems to occur when the previous page has been scrolled completely off screen. That is, one would read something like:

Page 2 line 2
Page 2 line 1
Document.odt — LibreOfficeDev Writer Document.odt - LibreOfficeDev Document editable doc
Page 1 line 50
Page 1 line 49

This is still better than before, and I think it most appropriate to close this issue as resolved and open a new issue for this behaviour.

Tested with LibreOfficeDev Writer version: 25.2.0.0.alpha0+ build f9b5dad09dbdd56ff064096695a946b03e9e2914 and NVDA alpha-33464,df6bd077 (2024.4.0.33464)

[michaelweghorn]

Thanks a lot for testing, @SaschaCowley.

The only bit of oddness is when moving backwards in braille, when entering the previous page, NVDA shows the the name of the edit control and then the window, before then moving to the last line of the previous page.

Do you know whether that is that related to something odd being exposed via the relations or something else, e.g. an extra focus/... event being sent for the document object?