Something about TriptychView and pre-rendering

[ClassWizard]

As we known, r2-navigator-swift uses the triptych view pattern to do some pre-rendering job. It looks good if both the previous and the next html contains many pages. Because it take some time for user to scroll to another previous/next html, so the reading system(WKWebView) can pre-render the new html pages during the time.

But it will not perform good enough when the html just contains a single page. If the user scroll quickly(probably by normal speed?), the reading system may not have enough time to pre-render. It will make the screen flash(blank content then loaded content) which is not very user-friendly.

I'm considering if there be some way can improve it. What do the reading system need to ensure the pre-rendering is having enough time. If user has enough pages to be read, reading system will has enough time to do pre-rendering job. So why not do we use a range of pages instead of triptych html?

For example, I assume that there should be 5-10 pages waiting for user to read. When the rest of pages is less than 5, (we can use the contentSize of the scrollView in next WebView) I initialize another webView to pre-render the next html. Then count the total pages. If not enough, do the same work again.

It just a preliminary idea. I don't know if that feasible.

[HadrienGardeur]

Well, to achieve the same effect with FXL files that rely on spreads, we'll need 6 webviews anyway (one per spread, two spreads per screen).

For pre-rendering, keep also in mind that the resource will be ready to be displayed faster once we move the user settings injection to the streamer. We'll avoid at least one redraw (when user settings are currently applied) compared to the current solution.

[ullstrm]

We're having problems with this aswell. There's sometimes a stutter when swiping/tapping to a new document. Also, when opening a document from the ToC, the stutter is almost always there.

What would be a preferred approach for how to solve this? Were would this fit in the roadmap?

[HadrienGardeur]

There's a real cost tied to pre-rendering in the background as well, especially for books with large resources. I think we could introduce more flexibility to the number of webviews that are used, but we need to be conservative about this.

@ClassWizard @ullstrm any of you on the Readium Slack or participating to the weekly calls?

It's a little unclear to me what you mean by "stutter". When we skip directly to a resources or first open a book, there will be some additional delay because the resource needs to be rendered. Is that what you mean or is it tied to a repaint?

[duongphamz]

We are having problems with this. We are using r2-navigator-swift to open epub file (comic), the html just contains a single image. When user scroll quickly, the reading system may not have enough time to pre-render. It will make the screen flash(blank content then loaded content). And our clients feel uncomfortable with this. Hope to get help. Thanks @llemeurfr From Gumi Viet Nam

[llemeurfr]

It seems to me there are 2 kinds of evolutions 1/ make so that more content is pre-rendered 2/ make so that the page appears when it is rendered (therefore no "stutter" of any kind). Solution 1 means more memory consumption, and heuristics for pre-rendering the content which may fail (e.g. if the users goes suddenly back and the 10 next pages already pre-rendered are unused). Solution 2 means that the user may wait a bit before the content appears, if he browses too fast for the pre-rendering mechanism to be effective. I would be in favor of solution 2 if we find a way to do it efficiently; what is your opinion?

[llemeurfr]

@judares, the fact that comics/mangas in EPUB must use an html container with a full screen image is a known issue for performance. You'll find traces of that issue in many articles, e.g. https://w3c.github.io/tokyo18-workshop/slides/simple-fxl-epub180910.pdf

Did you find an EPUB reader that is more efficient in that aspect than the Readium 2 iOS reader? if yes it wold be good to know how they do.

[llemeurfr]

@judares the ultimate solution we are studying is an evolution of the EPUB format, specialized for bd/comics/manga, where the html envelope is removed. This is under study at EDRLab & soon at W3C, and Readium software will support it next year. It may be a way to get higher efficiency after some months of production.

[duongphamz]

@llemeurfr Thank you for your support. We find an EPUB reader that more efficient in that aspect than the Readium 2 iOS reader is Folio Reader. https://github.com/FolioReader/FolioReaderKit Its reading system use Collection View to display FXL epub so there are no screen flash(blank content then loaded content) when user scroll quickly any more. Any problems if we use it with r2-lcp-swift ?

[duongphamz]

We tried to make a loading indicator when content is not loaded but our client do not want to wait a bit before the content appears when scrolling.

[llemeurfr]

@judares, you're free to use any piece of open-source software and re-assemble your app accordingly. The navigator is an independent piece of software in the R2 architecture, and you may even use alternative navigators depending the kind of content you're treating (this is what we do for e.g. CBZ content). Meaning that is you see that the publication is a manga (there is a need for some metadata in the content or generated at streamer level), you can select a different navigator than if it is text based.

Your finding is interesting, as we're working with the Folio team on the Android side at the moment for enhancing the Android navigator. When we focus back on the iOS navigator, we'll look at the Collection view solution.

[ullstrm]

Thank you all for your answers! With "stuttering" I mean that when quickly navigating to a new document that hasn't "had enough time" to layout - the text is misplaced for a brief moment, i.e. margins are all wrong, or the wrong content is shown (mainly when navigating backwards). It's as if the CSS has not done it's work properly yet.

I looked into solving this in the TriptychView and/or WebView, but instead came up with the solution of fading in the content with some delay after getting the documentLoad-callback from the JS. I didn't want to mess too much with the Navigator classes. For the end user - I think it's an acceptable experience.

I joined the slack earlier this week!

[HadrienGardeur]

Thank you all for your answers! With "stuttering" I mean that when quickly navigating to a new document that hasn't "had enough time" to layout - the text is misplaced for a brief moment, i.e. margins are all wrong, or the wrong content is shown (mainly when navigating backwards). It's as if the CSS has not done it's work properly yet.

I think this is somewhat different than what @llemeurfr is suggesting.

Initially, the addition of Readium CSS and then user settings caused repaints in the webview:

• the HTML was rendered without Readium CSS first
• a JS added Readium CSS, which triggers a repaint
• another JS injected user settings (CSS custom properties), which also triggers a repaint

The situation should be different now and do the following:

• Readium CSS is injected in the HTML by the streamer
• user settings are also injected in the HTML by the streamer
• HTML is rendered once with all CSS ready to go

I'm concerned that you're still seeing a repaint and I'm wondering if there's either something not properly implemented (ping @llemeurfr and @aferditamuriqi) or if this is the expected behavior (ping @JayPanoz).

Solution 2 means that the user may wait a bit before the content appears, if he browses too fast for the pre-rendering mechanism to be effective. I would be in favor of solution 2 if we find a way to do it efficiently; what is your opinion?

I looked into solving this in the TriptychView and/or WebView, but instead came up with the solution of fading in the content with some delay after getting the documentLoad-callback from the JS. I didn't want to mess too much with the Navigator classes. For the end user - I think it's an acceptable experience.

I think this is somehow different and necessary as well. In some ebooks, we might need to render resources that take a long time to render:

• CJK fonts for instance take a lot of time to render
• long resources with basic text layout (Proust for example) can also take a while
• resources with many large images or complex JS

For such cases, having an optional loading indicator that is displayed while the resource is not fully painted would be very helpful.

[JayPanoz]

I’m aware some (most? all?) implementations of CSS Multicol make two passes for layout but that’s accounted for in WebViews/Browsers – mind you, that’s pretty blurry since multicol is severely underdocumented @ the browser level, I actually discovered that in the infamous “Let’s remove CSS Regions in Blink” thread.

In ReadiumCSS, we’re forcing relayout through a couple of CSS hacks for the sepia and night reading modes, as not doing so will actually trigger nasty bugs in layout (e.g. margins in Firefox) or when it’s up to the GPU to handle rendering – e.g. filters and blend modes for images that, without those hacks, will simply disappear in Blink/WebKit. That shouldn’t create stuttering though as I paid hours of attention to that risk when designing and implementing those hacks, and they aren’t even visible when applying the setting.

Stylesheets are render-blocking as well so there’s probably something changing at runtime and which is triggering a style update. I had a similar issue to quickfix, same symptoms but doesn’t mean it’s the same evil. Anyways Chrome Dev Tools actually report a violation for relayout if that is the case and it’s fairly easy to check the trace and see what triggers it; Safari’s won’t so it’s really painful to debug if it’s not obvious.

[duongphamz]

@llemeurfr @HadrienGardeur Thank you all for your answers. After considering, we decide to make a loading indicator. Hope the iOS navigator will support comics/manga EPUB type soon. Thanks again.

[JayPanoz]

Oh… maybe we should take embedded fonts into account? There’s a lot of potential for messing first contentful paint there.

[HadrienGardeur]

@judares comics/manga are a different topic IMO:

• there's support for FXL in the current navigator
• there's also a separate navigator that's image-only and used for CBZ

For the use case that you're describing (comics distributed using FXL EPUB) there are two potential ways to improve the situation:

• you could detect at a streamer level that HTML is only used as a container and rewrite the spine/reading order to point to images directly and rely on the image based navigator
• we might want to default to 6 webviews for FXL (since spreads can cover 2 resources and we always want the previous/next spreads), which would also improve the situation for single spread FXL publications

[duongphamz]

@llemeurfr , @HadrienGardeur , Thank you all for your answers. Currently, With FXL we have a solution that we load image file from local server (after parse epub file by R2-Streamer). Then use it to display without using WebView. What is your opinion? Hope to get your answer. Thanks

[HadrienGardeur]

@judares there's something similar in the test app being used for CBZ.

The problem with FXL is that you can't be sure that it's just an image. A FXL resource (HTML) may rely on CSS for absolute positioning or SVG, all of which require a webview to be properly handled.

[mickael-menu-mantano]

I've reworked significantly the TriptychView to make it simpler in the fixes/triptych branch. It supports now pre-loading of an arbitrary number of resources (I guess it should be renamed now, triptych is ill-fitted). However, I don't have time to finish this so currently it is fixed at 1 before and after the current page, but you should be able to experiment with it.

Right now it's counting resources, but I think we should preload a number of positions (as in positionList) instead. This way we can handle seamlessly publications with both FXL and reflowable resources. And this would also be useful for short reflowable resources (eg. title page).

[mickael-menu-mantano]

I added the pre-loading of an arbitrary number of positions in the EPUB navigator. Since it's using positions and not resources, it not only works for FXL but also reflowable (for example a cover or summary pages amount for 1 or 2 positions, so several reflowable chapters might be preloaded).

This is part of the PR for the positionList, since I needed them. The amount of positions to pre-load is customizable through the new EPUBNavigatorViewController.Configuation struct: https://github.com/readium/r2-navigator-swift/blob/2d1a203ac6d62b4e9e2f4a8f7baf970a4e76c519/r2-navigator-swift/EPUB/EPUBNavigatorViewController.swift#L60