Not capturing the correct selected words for the highlight

[mrifni]

running the latest develop branch on a real device (oneplus 5)

please see the attached video

https://user-images.githubusercontent.com/1643992/133790542-6ec2d267-0116-47f2-af5f-faa8088a6065.mov

[mickael-menu]

Note to self: 0fdf1bffb52048bd849afa7cf698c4c1_sample.epub

[mickael-menu]

I investigated quickly, the saved text seems correct:

{
    "before": "to  manifest  it  over  all  religions,  although  the Mushrikoon hate it.ʺ [Sooratut-Tawbah, 9:33] Audio not supported!\nThe guidance is the beneficial knowledge, while the religion of truth is the ",
    "highlight": "righteous action",
    "after": "  that  is  based  upon  sincerity  to  Allaah  and  following  His Messenger (ﷺ).\nAnd beneficial knowledge includes any knowledge that contains a benefit to the Ummah for its life in the Dunyaa and"
}

Although we can see an interesting issue: "Audio not supported" is part of the highlight snippet even though it's not desired for user presentation:

<audio xmlns="http://www.w3.org/1999/xhtml" src="..." controls="controls">Audio not supported!</audio>

I'm not sure what's wrong with the righteous action snippet though, as even if I reduce the surrounding context to a couple of words, it is still not matched. Other words with duplicates are highlighted properly, even when the snippet contains the "Audio not supported" excerpt:

We might need to escalate this issue to Hypothesis, once this discussion is settled.

[og-abdul-mateen]

@mrifni but how are you even able to play the audio? I am trying to do the same, but couldn't find any documentation/sample that can do it.

[mickael-menu]

@og-abdul-mateen You don't have anything to do to support it. However on FXL EPUB it's currently broken because of this issue: https://github.com/readium/kotlin-toolkit/issues/131

[og-abdul-mateen]

That's interesting. I've been scratching my head over trying to support audiobook + text highlight support (basically Epub3 with MediaOverlay), but couldn't find a sample or documentation for it. Will try it on some Ebooks to see how it goes. Thanks @mickael-menu.

[mickael-menu]

Media overlays is not the same as <audio> resources embedded in HTML pages. They are not yet supported but will be later this year.

[og-abdul-mateen]

Umm okay. I have generated some Ebook files of my own using syncabook (plaintext + audio -> SMIL + XHTML -> EPUB3 with MediaOverlay), but as you mentioned, it might not work atm. Is there a spec for the supported <audio> embedded books format so that I can read more about it and maybe create my own versions using some tool?

PS: Sorry for bloating this thread with unrelated questions.

[mickael-menu]

I would suggest waiting for the Media overlays implementation, this is the most user friendly and easiest to generate. <audio> are just plain HTML elements that are displayed like in this page: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/audio The user must interact with it.

[og-abdul-mateen]

Do we have any ETA for that?

[mickael-menu]

Should be in 2023

[og-abdul-mateen]

@mickael-menu do you have sample Ebooks with audio that work with Readium Kotlin Toolkit on Android? I couldn't make any Ebook audio format work with Readium Android.

[qnga]

https://github.com/IDPF/epub3-samples/releases/download/20230704/cc-shared-culture.epub

This one is supposed to work. If not (with some patience), please report a detailed issue.

[og-abdul-mateen]

https://github.com/IDPF/epub3-samples/releases/download/20230704/cc-shared-culture.epub

This one is supposed to work. If not (with some patience), please report a detailed issue.

@qnga I tried this sample Ebook. Video works fine, but audio doesn't work. Even it would, this is not what I'm looking for. I guess I'll wait for the official MediaOverlay support in Readium Mobile. So far, Readium looks like the most modern Android library for Ebooks while ticking the right boxes for me (except for the MediaOverlay support).

[og-abdul-mateen]

@mickael-menu do we have any update on the EPUB3 with MediaOverlay support in Readium Kotlin Toolkit? If it's far away in future, can you suggest another solution that essentially does the same thing?

[mickael-menu]

It should be available in 2024, I can't be more precise than that.

I'm not sure about any alternative, it depends what you need and if you produce your own publications. If you must consume EPUBs containing media overlays, then no there's no alternative besides implementing it yourself.