search

krautzource / aria-tree-walker

A lightweight walker for labeled ARIA trees
https://krautzource.github.io/aria-tree-walker/
Apache License 2.0
2 stars 0 forks source link

support touch input #44

Open pkra opened 2 years ago

pkra commented 2 years ago

Never moved here from https://github.com/krautzource/mathjax-sre-walker/issues/9. To quote

On mobile, we lack proper exploration.

We seem to get only the benefit of being a progressive enhancement, e.g., Talkback users can switch to read-by-word and go through the description word by word. But we do not get full exploration and exhighlighting that way.

pkra commented 2 years ago

From https://github.com/krautzource/mathjax-sre-walker/issues/9#issuecomment-450821428

HammerJS seems inactive. Maybe https://github.com/AlloyTeam/AlloyFinger is worth a look (there are other libraries). The main question is if mobile AT has a way to pass inputs through.

pkra commented 2 years ago

https://github.com/john-doherty/swiped-events seems like a good source of inspiration.

pkra commented 2 years ago

I think I have a solution. Needs more testing because we'll have to listen (and preventDefault) for touch on the whole document.

pkra commented 2 years ago

Needs more testing because we'll have to listen (and preventDefault) for touch on the whole document.

Unfortunately, it looks like this idea won't work on mobile VoiceOver.

pkra commented 2 years ago

Unfortunately, it looks like this idea won't work on mobile VoiceOver.

There might be a chance after all. Needs investigating.

pkra commented 2 years ago

For future reference: https://patrickhlauke.github.io/touch/tracker/multi-touch-tracker-pointer-hud.html (double tap+hold+... should trigger this with VO)

pkra commented 1 year ago

Coming back to this after much pondering, I currently see two paths forward.

a) implement walking by swiping b) implement a new approach (more aligned with "standard" tree exploration, i.e., what APG shows).

For a), the main challenge is whether it's good enough (especially VO with touch would not be great UX) and if we find reasonable resolution for the up/down swipe vs scrolling conflict.

For b), I don't see a way to make this work (well) for non AT users. It would also require a fairly different bit of logic.

pkra commented 1 year ago

For b), links will finally come to haunt me.

pkra commented 1 year ago

Here's another stray idea while talking with @zorkow: a mini UI of 4 arrows, sticky on top-left (or wherever dir dictates).

pkra commented 2 weeks ago

Coming back to this after much unsuccessful research in the previous years.

Some notes

There are two problems and I don't see a simultaneous solution: touch input exploration vs exploration with mobile screenreaders.

For touch exploration, I have some leftover changes (which I don't fully understand right now) which do a decent enough job for touch users (except that it's hard to drop out of focus/interaction on safari but perhaps that's not insurmountable).

In theory, talkback users could use that approach since talkback allows gesture pass through. However, "4-finger-double-tap-and-hold + swipe" isn't exactly highly usable.

However, I couldn't get VoiceOver to work with that at all - the double-tap-hold-wait-for-three-dings-interact never worked for me. And again, it's not exactly usable.

On the other hand, the approach from https://github.com/krautzource/granularity-walker works for talkback and VO users. Perhaps that approach could work for touch input as well. Perhaps both could be combined.