search

ankidroid / Anki-Android

AnkiDroid: Anki flashcards on Android. Your secret trick to achieve superhuman information retention.
GNU General Public License v3.0
8.28k stars 2.17k forks source link

talkback can't see sometimes front, sometimes back and sometimes both sides of anki cards during a study session. #14934

Open nick777999 opened 8 months ago

nick777999 commented 8 months ago

Checked for duplicates?

What are the steps to reproduce this bug?

  1. enable talkback
  2. open ankidroid as usual (or doesn't matter if you do the steps 1 and 2 in refersed order)
  3. touch the deck to start reviewing material
  4. swipe from left to right with one finger on the screen and try to get to the card content. you could also touch the screen with one finger directly on the content or drag it there.
  5. if swiping, you should hear webview just a one swipe away from the actual card content (which is an expected behavior since a screen reader reads windows).
  6. at this point, swipe once more and you should, ideally, hear content spoken.
  7. do it several times and you will definitely encounter this issue. especially if you do it really fast like some blind people like me, do it all the time.

Expected behaviour

at the step 6, content should be read by a screen reader all the time.

Actual behaviour

sometimes (very frequently) at the step 6 screen reader is not able to read the contents.

Debug info

AnkiDroid Version = 2.16.5 (953b9bd879269910ee962b520da9705336d8bc2d)

Android Version = 14

ProductFlavor = play

Manufacturer = samsung

Model = SM-S911B

Hardware = qcom

Webview User Agent = Mozilla/5.0 (Linux; Android 14; SM-S911B Build/UP1A.231005.007; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/119.0.6045.193 Mobile Safari/537.36

ACRA UUID = eba7efdf-bc86-4b88-aafc-234e4a89c124

New schema = true

Scheduler = std3

Crash Reports Enabled = true

DatabaseV2 Enabled = true

(Optional) Anything else you want to share?

when using ankidroid with screen reader, there is some serious discomfort using it. specifically with card studying and webview. I am using a talkback with the latest android available and sometimes during studying screen reader suddenly can not see front or back sides (sometimes both of them) of the card, as a result i can not see what the card is asking. it only says webview and that is it. that frequently leads the ankidroid crashing. i have tried to analyze behavior by keeping track of the length of the card, frequency of the error occuring, waiting before i answer to allow on the one hand a webview to load what it needs to and on the other, a screen reader to parse how it needs to do it. Generaly, since there is less overhead involved because network latency is not there, i suppose this should be an easy and quick process, plus, I don't use much html in my cards, only plain text (if anki doesn't do something with it). tried to uninstall webview updates, but no success. this action was suggested by an android system itself, by showing a message like: an updated version of webview caused ankidroid to crash. please uninstall it and try again. no corelation was found whatsoever. I have tried exploring the screen by touch, just to move the focus on the webview, no success, neither webview, nor the card itself is visible (for screen reader) hence nothing gets focus. even worse, when the screen reader sees the card, contents and even a webview window can not be focused. only the show answer button or if the back side is not visible as well, answer buttons are visible at the bottom. Also tried clearing the cache of ankidroid and a webview, uninstall and reinstall, still happening. sometimes app closing from the app drawer and relaunch helps, sometimes it doesn't. As a last resort, I tried with built in mechanism for tts. that kinda worked, but because a screen reader was focusing on the back button at the top (which is normal and necessary), speech was interrupted and there was nothing to do instead of opening up more options and pressing the play button. This has some other issues, such as speed control, as I usually read material at a very high speeds and speech output is very slow as well as system was asking me which language was used on the card. providing it manually for thousands of cards in several decks is just a very time intensive task which we should use to make results better . sometimes when you press a show answer button, i can not get to that button, because of ankidroid crashes. There is no difference between enabling new backend or experimental features in this regard, as I have tried both. I have never experienced issue like this in any app, for instance gmail uses webview, but have been using it great ever since. That's what caused me to look into this deeper and submit a bug report. This is as far as I could get in my explorations. if that was windoes, I would share some detailed info about objects, accessibility properties, screen reader perceptions of the presented material and even could suggest what would work, but since I don't develop for android platform, I'm sadly not able to do that. if there is anything i can do to help you better understand the issue at hand, please don't hesitate to let me know and i will do my best to get back. I would rather not switch to something else, because I love the community, available documentation and that it's opensource product, not to mention additional time of research what is accessible and what's not, card formats, transfer issues of thousands of cards, checking accuracy and etc etc.

if there is any detailed logging facility within your app to gather logs and provide for further analysis, please let me know so I can also attach them. meanwhile I am attaching a debug info here. thanks so much in advance.

Research

welcome[bot] commented 8 months ago

Hello! 👋 Thanks for logging this issue. Please remember we are all volunteers here, so some patience may be required before we can get to the issue. Also remember that the fastest way to get resolution on an issue is to propose a change directly, https://github.com/ankidroid/Anki-Android/wiki/Contributing

eginhard commented 8 months ago

Thanks a lot for your feedback @nick777999. We haven't had much feedback from screen reader users before (probably because the app doesn't support them very well as you noticed), so this is very useful. I'll also link this issue for context, although it doesn't contain much that you didn't already mention: #7913

I am actually not able to get TalkBack to read any of the webview contents at all when testing it (I'm on version 2.17alpha8 but I think it was the same before). For me the content is always just announced as "unlabelled", so this is definitely something that needs to be improved.

Regarding TTS, it is known that it conflicts with TalkBack, see #6369. However, TTS support is currently also being worked on by @david-allison to use the same tags as Anki Desktop, so this will need some reevaluation anyway. One advantage of this is that in a future version modifying TTS speed will be possible if supported by the selected voice (like this: https://docs.ankiweb.net/templates/fields.html#text-to-speech-for-individual-fields).

One thing I was wondering about is how TalkBack handles foreign languages. For example, if you have English on one and French on the other side of a card, would TalkBack be able to pronounce these correctly? The advantage of TTS here would be that you could define the languages in the card template, so it can use an appropriate voice.

nick777999 commented 8 months ago

Hello again! First of all, Thank you @david-allison and @eginhard to getting to this issue so quick. I appreciate your time and efforts to dedicate your resources and energy as volunteers. to answer a question regarding how would a TTS pronounce the certain languages, it depends. for instance, I love to use espeak. It is free, open source, supports a lot of languages, retains quality of sound on very high speeds. disadvantage: a robotic sound. still, I recommend BTW, when I have used a tts, it defaulted to espeak. To generalise my answer, that would depend on the speech synthesizer and how it handles languages. It might have a mode to automatically switch to the necessary language, or it might be controlled by the screen reader application settings. The idea that you, @eginhard, suggested, is alredy beeing utilised in other applications, such as quizlet. That is useful not only for blind people, but for sighted folks as well. speaking about the "unlabeled", that hypothetically tells me 2 things:

  1. you are using one of the latest versions of talkback that has image descriptions built in;
  2. when putting your finger over the card content, there should be an image or something like this. Actually, for me that output is the same. I was wondering, if there is some image or not, because that usually happens when you touch a part of the screen that contains an image. Ideally at the place of the card there should not be image, if it was not purposefully inserted there. I would kindly ask you to confirm this one. I was brainstorming the ideas and doing some comparisons over the day and here are my results so far. when loading a card and by carefully inspecting the screen reader output, I got some impression like after triggering one of the answer buttons, the whole window reloads and loads the new card in webview widget. I have this impression based on 2 things: screen reader has some delay and it focuses on the back button at the top left side of the screen as on first run. Next swipes right from there (which is a command to move to next element) take me to app sidebar buttons and that proved hypothesys. if only focus movement had taken place, there would be minimum delay, but that delay might be a window load in necessary systems. If that works like so, then it would need to initialize webview on every reload and there might be some underlying issue that might be causing it. Contrary to that, I have checked on ankiweb.net with chrome (which, by the way, also uses webview). there, after you answer a card, the website does not reload and what happens is that focus is beeing moved on show answer button. the prove here is simple, if a chrome page reloads, you would hear a percentages bar spoken by talkback or beeps emitted. you can test it yourself and see the performance is much faster. If that's true and it works like so, my idea is to have a
    or a
    where we would insert a user content and every time we need to load the next card or something, we'd just replace the content of the necessary containers and that's it. Maybe with javascript or some other technology. this way, with this section we'd control the positioning of front and back sides as well as retaining the content agnostic nature of the application and a user formatting of their cards. I guess that might be decreasing performance overhead as well a little, as system will not be required to initialize a whole window from scratch again with necessary parameters. BTW, that had a performance boost in card statistics as well. If on ankidroid I could do a 4 or 5 cards per minute at full speed, on slow speeds while testing ankiweb with chrome, I could do somewhere around 6-7 cards per minute. This one would require testing before murging to stable release, maybe in alfa or beta channels first. another option, although very simple but temporary for text only cards, is to use a textbox widget which just displays a text. So the current window model will stay the same, but instead of webview it would just display the content in textbox. I imagine this beeing configured from the settings window.

p.s. If anyone would try to test the points mentioned above, don't hesitate to let me know and I will put a steps to reproduce and what the expected result should be. better educational apps means better education for everyone and I believe based on my practice, better testing results in better systems, so I'm opened to requests if anyone needs those steps to verify. Thanks again.

david-allison commented 8 months ago

Related issues

nick777999 commented 6 months ago

Hello. Are there any news regarding this? I found that pressing on more info and then pressing back button reloads webview.

wyy511511 commented 5 months ago

Hello. Are there any news regarding this? I found that pressing on more info and then pressing back button reloads webview.

Hi Nick, I care about accessibility issues and I appreciate your detailed report and followup! I am trying to reproduce, understand, and consider each point you mentioned. I will start by addressing the issues with unlabeled images and pure text cards.

  1. Resolving this issue on basic pure text cards I believe rethinking the logic of webview may be a large issue, but don't worry, based on my limited knowledge and skills, I believe the pure text part can be addressed. Anki is open source, so at the very least, I can compile a version of the APK for you to use.

  2. Example of language switching in Quizlet? Can you provide specific examples of language switching in Quizlet? For instance, a link to a set of Quizlet cards would allow me to experience and analyze Quizlet's specific implementation.

  3. Understanding and everyday experience with the latest multimodal Generative AI Additionally, I would like to ask about your understanding of the latest AI technologies. Simply put, chatbots are now very intelligent. Are you familiar with the development of ChatGPT by OpenAI? Or perhaps Microsoft Bing, which is free to use, or Google Gemini (previously called Bard)? I think these technologies may reshape the way blind individuals interact with the outside world. The latest version of BeMyEyes utilizes basic functionalities of these technologies (but I believe they can have greater potential in terms of accessibility). If you're willing to share any experiences you have in this regard, I'd be interested to hear them!

nick777999 commented 5 months ago

Hi @wyy511511 ! Thanks for your followup, time and commitment. I very appreciate it. Starting with AI, yes, I am aware of them and use them very well for general accessibility. Speaking of language switching, I have to look the context up again a bit and get back to you, since I have left quizlet as well and ended up making my own mini version of those tools with simple html and javascript as well as back end for db operations and stuff. I will, however, of course cooperate as much as possible to make this wonderful place a bit accessible.

wyy511511 commented 4 months ago

Hi @wyy511511 ! Thanks for your followup, time and commitment. I very appreciate it. Starting with AI, yes, I am aware of them and use them very well for general accessibility. Speaking of language switching, I have to look the context up again a bit and get back to you, since I have left quizlet as well and ended up making my own mini version of those tools with simple html and javascript as well as back end for db operations and stuff. I will, however, of course cooperate as much as possible to make this wonderful place a bit accessible.

Hi Nick, Today I reproduced the issues you mentioned. Indeed, it's not user-friendly for blind and visually impaired individuals. 1."Unlabelled" is not related to images; my test decks consist purely of text. However, the repetitive reading of "webview" and "unlabelled" greatly affects the user experience.

2.Distinguishing between the front and back of cards is difficult because I have to rely on swiping left and right to change/switch line. Initially, the last line of text on the back of the card is read, and there is no audio prompt when switching from the first line of text on the front to the corresponding line on the back.

  1. The language switch works fine in my tests with decks containing Chinese and French cards, while the system language is English (with AnkiDroid buttons in English), without requiring additional settings.

4.TalkBack is unable to read the content of the second and third cards aloud; further testing is needed.

5.Floating widgets from other applications affect the program sequence (I will examine the TalkBack source code to see how screen content is traversed).

6.Additional testing I plan to conduct includes:

Continuing to reproduce and summarize problematic scenarios, conducting in-depth testing of multimodal card decks containing audio and images.

7.Rethinking the interaction design, especially how blind users can get started. I would like to ask: how important is card editing to you? Are you using cards you created yourself or downloaded from the community? I'm considering creating a sample card deck as an accessibility tutorial. This could help beginners understand card structures and basic editing and usage operations.

No worries for Quizlet example. Thank you for your feedback!

Todo: I need to revise my reply to make it more organized and easy to understand.

github-actions[bot] commented 1 week ago

Hello 👋, this issue has been opened for more than 3 months with no activity on it. If the issue is still here, please keep in mind that we need community support and help to fix it! Just comment something like still searching for solutions and if you found one, please open a pull request! You have 7 days until this gets closed automatically