Make the app more accessible with a screen-reader

[gnprice]

We've just completed #4232 with #4355, by adding captions to two particularly critical bits of the UI. But there are still more things we should do to make using the app a good experience with a screen reader.

The essential thing for working on this issue is to take some time using the app this way -- and before that, to take some time learning how to use one of these systems, if you're not already a regular user of them. Here's the upstream guides for doing so with either Android or iOS: https://developer.android.com/guide/topics/ui/accessibility/testing https://developer.apple.com/documentation/uikit/accessibility_for_ios_and_tvos/supporting_voiceover_in_your_app

In particular, the following steps are highly educational, and essential for working in this area:

• Take your actual physical device, either Android or iOS.
• Turn on its screen reader. (On Android this is called TalkBack; you may have to install it from the Play Store. On iOS it's called VoiceOver.)
• Go through the tutorial the system provides for how to use the screen reader and how to navigate around the device using it.
• Then leave the screen reader on, and keep relying on it as you try to do things on the device (including exploring the screen reader's own settings.) This way you'll keep learning more about how to use it in practice. (You might keep a quick-reference page open to remind yourself of the gestures, to help learn them faster.)

When I did that for the first time, on Android, I spent about a half-hour and learned quite a lot. As you continue working in this area, you'll keep doing more of it.

One general principle in choosing how a given widget should work is that we want things to feel natural and as-expected for people who regularly use one of these systems. So the labels we choose, and other aspects like what's selectable, what order things come in, etc., should match what other apps do, especially apps from the platform maker (Google or Apple) and other major apps.

Here's a list of some of the things we should improve or investigate (in the order I found them, not necessarily priority):

• [ ] The up/back button (NavBarBackButton.js) gives the right label for Android, but it should probably be something else on iOS.
• [ ] In general, we haven't yet spent time trying iOS's VoiceOver, either in general or with Zulip. It would be valuable for someone to do that -- up/back will probably not be the only thing that should be different there vs. Android.
• [ ] I don't know if the onboarding flow (logging into Zulip, the first time you open the app) works well or at all. We should try it, find issues, and fix them.
• [ ] The pick-account screen has "trash" icons for deleting an account. These don't seem to work in TalkBack.
• [ ] The streams screen doesn't say "in list" when you enter the list of streams (in TalkBack/Android). Based on other apps, I think it should. I'm not sure how to properly give TalkBack the hint that the streams are a list.
• [ ] On the streams screen, when you select a stream it says something like "Test here. Five thousand seven hundred and five." (for #test-here with 5705 unreads.) The number should probably say "unreads" or something. Needs playing with other apps to see what conventions are.
  • [ ] Same thing on the unreads screen, for a stream or topic etc.
• [ ] When you enter a screen, there's no announcement of what the screen is. E.g. when you navigate to a narrow, we should say the stream and topic, or that it's a PM thread with such-and-such people, etc.
  • [ ] Similarly, when you come back to Zulip from another app, via the overview screen, we don't announce what screen of Zulip you're on, and we should.
• [ ] In the nav bar on a stream or topic narrow, the # is selectable. It shouldn't be.
• [ ] In a topic narrow, the up-arrow nav button has no caption.
  • [ ] Probably the same is true of all our nav-bar buttons, except the up/back button which we just fixed in #4355.
• [ ] In the message list, the avatars are selectable. They probably shouldn't be -- they don't carry any information that isn't in the sender name, which comes right after.
• [ ] In the message list, maybe the sender names should say "Sender: ${name}" instead of just the name? And the sent times say "Sent:" or "Sent at:"? Not sure -- needs playing with other messaging apps to see what conventions are.
• [ ] Also maybe the navigation should move a whole message at a time? Again, needs playing with other apps.
  • Aha, here's at least what the stock Android Messages (i.e. SMS etc.) app does: there's one item for the whole message, and one for the body. The one for the whole message reads out something like "You said: Blah blah blah. Dec 9, 00 colon 46, read" -- so it starts by saying the sender, then reading the body, then saying when it was sent, finally whether it's read or unread.
• [ ] In the compose box, the "plus" icon (to open the compose menu) left of the input box has no caption. Same for all the buttons inside it.
• [ ] In the compose box, if you open the compose menu then the focus goes all the way back to the top of the screen. Probably it should stay there at the compose menu. (And probably there should be an announcement that you've opened it?)
  • Or: maybe the compose menu shouldn't open at all? And instead, we'd have a "local context menu". (At least on TalkBack.) That seems to be the way some other interactions are handled -- e.g. if you have a notification (on Android), you can do a TalkBack-specific gesture to pull up the "local context menu", which is a modal designed for efficient non-visual interaction, and use that to expand or dismiss the notification. Not sure -- experimentation needed. No, I think using the compose menu normally is the right path. That's e.g. what stock Android Messages does, and it has a fairly similar compose menu.
• [ ] In the message list, if there's a reaction, sometimes it works great (saying the name of the reaction), and other times it doesn't (saying a number like "one hundred and twenty-eight".) Instead, we should always say the name of the reaction. My guess is that the variable determining which are affected is Unicode emoji vs. realm emoji. (When fixing this, also test the :zulip: emoji, which is the only example of a third kind.) Needs debugging and then fixing.
• [ ] In compose, at the "Send message" button, it might be good to add a reminder of where you're about to send to.
  • I noticed this in the stock Messages (i.e. SMS etc.) app on Android -- selecting that button caused it to say "Send message" and then the name of the person.
• [ ] On the home screen of the app, none of the nav icons at the top have captions.
• [ ] On the PM-conversations screen, the avatars are selectable. They shouldn't be -- they don't add anything over just the conversation as a whole.
  • [ ] Same thing on the "recipients list" screen you get by hitting the "info" icon at the end of the nav bar when looking at a group-PM thread.
• [ ] On the PM-conversations screen, if there's a group-PM thread, the initials that we use for making up an avatar get read out. They shouldn't be.
• [ ] In a group-PM narrow, the nav bar shows the users' avatars but doesn't give their names. Instead, when you select one of the avatars it should read out the user's name.
• [ ] In a chat narrow where the compose box is visible but not opened, the touch target to focus it is too small. (I think this is an issue even for sighted users, but I'd never quite noticed it until using TalkBack -- I had trouble trying to select it several times in a row, so much that I thought I must have gotten things into a bad state that indicated some worse bug.) Specifically it's only as high as the visual box of the input itself. Instead, it should include the space above and below it, to the top and bottom of the compose box. (This is what the "plus" and "send" icons next to it already do, for example.) In particular, that would mean that if you tap at the very bottom of the screen, you'll open it, instead of getting a little "nothing there" error beep.
• [ ] … (Surely more! Play around with the app and find them, and add comments below.)

[gnprice]

For anyone who regularly uses a screenreader, I'd be especially interested in your feedback on what's most annoying for you in using Zulip, or thoughts on how a specific part of the UI should work, or any other aspect of this that you have something to say about :slightly_smiling_face:

[gnprice]

Another note: the old PR #2753 contains fixes for some of the items above. In particular:

• On the PM-conversations screen, if there's a group-PM thread, the initials that we use for making up an avatar get read out. They shouldn't be.

• when you select a stream it says something like "Test here. Five thousand seven hundred and five." (for #test-here with 5705 unreads.) The number should probably say "unreads" or something. Needs playing with other apps to see what conventions are.

• most, possibly all, other nav buttons on main screen and in nav bars

• On the PM-conversations screen, the avatars are selectable. They shouldn't be

• different message for NavBarBackButton between iOS vs. Android

A good place to start would be to revive the changes from that PR.

[Gautam-Arora24]

Since I have already worked on this type of issue earlier where I added accessibility labels to the send and back button. I would like to work on this issue as well.

[gnprice]

@Gautam-Arora24 Sounds great -- please do!

[chrisbobbe]

• [ ] In the nav bar on a stream or topic narrow, the # is selectable. It shouldn't be.

Hmm, could we revisit this a bit? I think that icon usefully conveys some important information, and it might make sense to keep it as a selectable item (but give it an accessibilityLabel with that information if we do). If we don't make it selectable, I'd still like to convey the information, but we'd need to find another way to do it. (Perhaps by making a single selectable item out of the parent View of the StreamIcon and the Text with stream.name, and give that an accessibilityLabel that includes the information plus the stream name.)

That information is:

• we're looking at a stream or topic narrow, not a PM narrow
• the stream is muted, or not muted
• the stream is private, a.k.a. "invite_only", or not

[chrisbobbe]

Also, I notice that the subscribe/unsubscribe switches in the "All streams" tab are impossible to interact with, at least with VoiceOver on iOS.

[Gautam-Arora24]

Also, I notice that the subscribe/unsubscribe switches in the "All streams" tab are impossible to interact with, at least with VoiceOver on iOS.

@chrisbobbe The subscribe/unsubscribe switches are impossible to interact with just because the Touchable component in StreamItem groups its children into one accessible component. Therefore, when we use VoiceOver for a list item in the "All Streams" tab, it doesn't individually select a Switch. What are your views on this?

[chrisbobbe]

On iOS, I notice that in the compose box, the order of selection goes like this: "expand menu" button, "send message" button, then after that, the message-content input field. It should go from left to right: "expand menu", message-content input field, "send message" button.

Probably the misuse of FloatingActionButton for the "send message" button is at least partly to blame. The button isn't (and shouldn't be) "floating".