search

dictation-toolbox / Caster

Dragonfly-Based Voice Programming and Accessibility Toolkit
https://dictation-toolbox.github.io/Caster/
Other
340 stars 121 forks source link

Integration with OS accessibility APIs #814

Open LexiconCode opened 4 years ago

LexiconCode commented 4 years ago

Issuehunt badges

@lexxish @alexboche

Is your feature request related to a problem? Please describe. Put simply re-implementing Dragon's Select-and-Say capability and text navigation.

Describe the solution you'd like Work to explore tool that use accessibility API. Key concepts

Accessibility APIs There can be overlap between applications with the respective OSs. I have only research tools for exploring the Windows OS

Other APIs Accessible Rich Internet Applications (ARIA)

Scintilla

MFC classes to encapsulate the Scintilla edit control

Java Access Bridge is for Windows OS interface with Java software https://doc.bccnsoft.com/docs/jdk7-docs/technotes/guides/access/jab/introduction.html https://docs.oracle.com/javase/7/docs/technotes/guides/access/enable_and_test.html https://docs.oracle.com/javase/7/docs/api/javax/accessibility/package-summary.html https://github.com/nvaccess/nvda/blob/defed0ef2317cf3192922692a40cc2eb92d41143/source/JABHandler.py

Describe alternatives you've considered

Additional context

Dragonfly as a Accessibility API which is OS/controller agnostic, Due to the pyia2 that exposes Google Chrome, Firefox and Gitter(electron based on chrome) is limited.

The Dragonfly accessibility if you can be leveraged by this Caster grammar accessibilityapi.py

Accessibility and Document Object Model (DOM) API Comparisons accessibility.linuxfoundation.org/a11yspecs/atspi/adoc/a11y-dom-apis.html

How to proceed

I've been looking into the following as many of the accessibility API UI automation platforms.

WinAppDriver robotframework pywinauto Python-UIAutomation

pyatspi2 for AT-SPI

The next step would be experiment integrating APIs and frameworks into Dragonflys Accessibility API. Staying in line with its design principles.

IssueHunt Summary ### Backers (Total: $200.00) - [itsrainingschmen itsrainingschmen](https://issuehunt.io/u/itsrainingschmen) ($200.00) #### [Become a backer now!](https://issuehunt.io/r/dictation-toolbox/Caster/issues/814) #### [Or submit a pull request to get the deposits!](https://issuehunt.io/r/dictation-toolbox/Caster/issues/814) ### Tips - Checkout the [Issuehunt explorer](https://issuehunt.io/r/dictation-toolbox/Caster/) to discover more funded issues. - Need some help from other developers? [Add your repositories](https://issuehunt.io/r/new) on IssueHunt to raise funds.
caspark commented 4 years ago

I cobbled this together as an example after @LexiconCode asked me about UI Automation after I mentioned it in the dictationtoolbox general channel: https://gist.github.com/caspark/36b1f2b47e37735e323863b1dda7458f

It's just a quick demo of fetching text from the control under the cursor.

LexiconCode commented 4 years ago

Some notes regarding using uiautomation PIP package.

LexiconCode commented 4 years ago

Application Evaluation

Successful - UI Automation

These are applications that bear minimum of enabled grep text from various fields

Unsuccessful

These are applications that have been unsuccessful grab text for various reasons. Some of these applications may respond better to IAccessible2, scintilla or other APIs. Further research is required

isidorn commented 4 years ago

Hello, VS Code dev here,

Please note for vscode accessibility support on should only be required on Linux. On windows and mac in theory it should work just fine with auto. Also note this issue https://github.com/microsoft/vscode/issues/98256 Which just fixed that it is possible to set caret selection on vscode on linux and mac. However this is only availalbe in vscode insiders in a couple of days. So in general I always recommend to use vscode insiders.

And let me know if we can help somehow from the VS Code side.

LexiconCode commented 4 years ago

Python-UIAutomation and pywinauto do not properly support functionality to fully grab the necessary information like carrot absolute/relative and so on.

This means were limited to improving 1 of the 2 above projects. Other less explored are leveraging screenreader technologies. screen readers use the same information needed for text manipulation and are often implemented using OS specific accessibility APIs.

Orca or Nvda nvda repo on GitHub https://github.com/nvaccess/nvda Orca repo https://github.com/GNOME/orca

LexiconCode commented 2 years ago

I will update this thread this week there's been some significant changes.

Accessibility and Document Object Model (DOM) API Comparisons https://accessibility.linuxfoundation.org/a11yspecs/atspi/adoc/a11y-dom-apis.html

issuehunt-oss[bot] commented 2 years ago

@itsrainingschmen has funded $200.00 to this issue.