Improve highlights, especially in night mode

[JayPanoz]

I'm submitting a feature request (raised from a bug report during yesterday’s engineering call).

Short description of the issue/suggestion:

Highlights module has been kind of a placeholder for a long time as we never discussed those properly in the past, so not many people use it, and it’s not even compatible with night mode.

The selector isn’t even customisable like most others in src.

Steps to reproduce the issue/enhancement:

N/A.

What is the expected behaviour?

This module should at least be updated to solve the two issues above, but the opportunity to have a “standard” for all Readium-related apps is perhaps even more important there, as it could offer a good-enough default for a lot of apps.

What is the current behaviour?

See description.

Do you have screenshots, GIFs, demos or samples which demonstrate the problem or enhancement?

N/A

What is the motivation / use case for changing the behaviour?

Provide better defaults for highlights, in a way that makes sense for implementers.

Do you know which CSS modules (stylesheets) are impacted?

• highlights
• night mode

Please tell us about your environment:

• Platform: all
• Browser / Rendering Engine: all

[danielweck]

The current prototype in r2-navigator-js utilises a RGB colour value (triplet) provided programatically by the API consumer of the "highlights" component (the default colour is red-ish). An opacity level is automatically added by the component (i.e. default translucency, plus an alternative value for mouse hover). In other words, API consumers can only set the colour, not the transparency values. The same colour and opacity are used regardless of the ReadiumCSS "theme".

[danielweck]

So, the current ReadiumCSS draft proposal is to use CSS class names: .readiumCSS-[yellow|green|orange|pink]-highlight

https://github.com/readium/readium-css/blob/master/css/src/modules/ReadiumCSS-highlights.css

This will not be usable in practice with the r2-navigator-js prototype, because the aforementioned RGB triplet is applied via CSS rgba() (added opacity as per the two aforementioned translucency levels) on various elements that make up the whole "highlight" visual object made of several rectangles (HTML or SVG), using different CSS properties depending on context (background, border, outline).

[JayPanoz]

Yeah I’m wondering whether we shouldn’t even get highlights out of ReadiumCSS now, given this also reminds me of the Highlight API doc.

And more generally, it will be a lot more complex to handle/customise if it’s in ReadiumCSS.

[danielweck]

Yes, ReadiumCSS is by definition "opinionated" (that is what makes it useful), so at best ReadiumCSS could define highlight colours that "work" with its own predefined themes. That being said, if the styles authored at the source in EPUB content documents deviate from the typical "back text on white background", then all bets are off, and it is not possible to guarantee acceptable contrast levels / readability. However, it would be nice if all R2 apps that implement support for "highlights" offered the same choice of colour palette. In other words, the RGB programmatic API in r2-navigator-js currently allows 256x256x256 possibilities, but this could be restricted to an "opinionated" set of triplets, maybe defined at the ReadiumCSS level?

[JayPanoz]

Sorry had to sleep over this following your remarks.

A few thoughts. Overall, I think there is a possibility I align the module with all others, while providing defaults, and extensibility.

the current ReadiumCSS draft proposal is to use CSS class names: .readiumCSS-[yellow|green|orange|pink]-highlight

It’s really still more of a placeholder than a draft at this point so classes here are more like references making “highlight colours” obvious.

That also means this module is completely different than all others in its design right now. For instance, a RGB colour value (triplet) could become 4 CSS variables (one for R, one for G, one for B, and one for alpha), as seen in this article for HSL(a) for instance.

But, as you highlighted…

using different CSS properties depending on context (background, border, outline).

That could be handled via a “type” flag probably, once again as a CSS variable.

That being said, if the styles authored at the source in EPUB content documents deviate from the typical "back text on white background", then all bets are off, and it is not possible to guarantee acceptable contrast levels / readability.

Just as a note because I haven’t had the time to read it thoroughly but there are attempts to handle text based on the background-color e.g. https://css-tricks.com/css-variables-calc-rgb-enforcing-high-contrast-colors/ (there are also the mix-blend-mode that can help, cf. end of article, but maybe that would introduce some weirdness in the form of text being inverted.)

this could be restricted to an "opinionated" set of triplets, maybe defined at the ReadiumCSS level

So I think there is a possibility of having opinionated set of triplets while allowing for customisation. I’ll try coming up with a proposal asap so that it’s clearer but, something like (pseudo CSS code)…

:root {
  --r: 255;
  --g: 255;
  --b: 0;
  --a: 0.5
}

*[style~="readiumCSS-highlight-bg"] {
  background-color: rgba(var(--r), var(--g), var(--b), var(--a));
}

*[style~="readiumCSS-highlight-border"] {
  border-color: rgba(var(--r), var(--g), var(--b), var(--a));
}

Knowing that if you had other values as inline styles on the element itself, those inline values will be used for background-color, etc.

<div style="… css-variables"></div>

Not a silver bullet, mind you, but at least that would offer defaults + could still be used dynamically if the RS hasn’t implemented the feature yet – I’m really fine with JS implementations for such features, given CSS is lacking for such complex issues.

[danielweck]

Yes, this looks like a step in the right direction. Thanks Jiminy.

[JayPanoz]

Related notes:

• picking between rgba and hsla would probably be a plus, given the latter has become quite a popular choice with CSS custom properties so should be an --RS-highlight-mode: rgba | hsla that can be appended to the :root element or something – with alpha a common css var;
• it should also probably handle svg out of the box.

Well at least now I have a clearer idea how the change will turn out, and it’s not massive work – documenting it will be the heavier task.

[JayPanoz]

Also for reference: styling + SVG handling in r2-navigator-js

[mickael-menu]

Yeah I’m wondering whether we shouldn’t even get highlights out of ReadiumCSS now, given this also reminds me of the Highlight API doc.

At first glance the Android highlight implementation is not using Readium CSS's highlight classes, and I also wonder if it should be part of Readium CSS or not. To align platforms, we can use a shared specification with a set of default colors, but I would expect reading apps to want to customize these colors.

If there was a way to have different color variants per reading mode in Readium CSS, it could make it worthwhile though.

[JayPanoz]

So let’s use this comment for voting on the removal of this module in ReadiumCSS.

in this mobile issue, @mickael-menu said:

After thinking more about this, I believe that highlights should not be in the scope of Readium CSS. If we have some default highlight colors in Readium, they should maybe be documented in architecture, and declared in the test apps. There's just too many different ways to implement highlighting, and also we couldn't reuse the colors declared in Readium CSS for other formats such as PDF.

We also know the Thorium implementation doesn’t use ReadiumCSS, and have this r2-navigator-js issue re. dev needs, which can inform our decision.

What’s become clearer is that ReadiumCSS would be way too constraining implementation-wise as you would have to rely on what it provides as CSS variables for hsl(a)/rgb(a), and it might well be cumbersome given how many additional variables it would create.

In addition, we shouldn’t forget other formats – PDF as mentioned by Mickaël but I’m assuming web publications could have TTS with sync/overlaid highlights, notes, etc. as differentiating features from web browsers.

If you’re in favour of removal, use 👍 . Otherwise, use 👎

[danielweck]

Further data points to consider - in r2-navigator-js there are now 3 "states" for a given highlight: 1) normal display (for example: 50% opacity) 2) during mouse hover (for example: 60% opacity, i.e. more emphasis) 3) during mouse hover on other highlights (for example: 40% opacity, i.e. less emphasis so that overlapping highlights are easier to visually-identify)

Furthermore, CSS mix-blend-mode = multiply is used to ensure text is legible despite the transparent overlay, and this is applied to an "abstract" container div as opposed to "concrete" individual client rectangles (which are optimized to prevent duplicates, overlaps, jagged edges, etc.).

Without going into too much detail, you can see that there are several optimization paths to achieve decent performance when dealing with thousands of character-level drawings (e.g. to present search results inside HTML documents). These techniques make it hard to standardize on simple ReadiumCSS styles / rules.

Note that we use the same CSS styling techniques for both SVG and HTML render modes.

Also note that "highlights" in r2-navigator-js include "strike-through" and "underline" as well as the "background colour" type of visual annotation.

[JayPanoz]

Ah thanks for these details. If ReadiumCSS had to handle all of this, that would turn this module into a labyrinthine system indeed so it’s not worth bringing confusion into an already tricky issue.

[mickael-menu]

during mouse hover on other highlights (for example: 40% opacity, i.e. less emphasis so that overlapping highlights are easier to visually-identify)

That's smart :) I wonder if a similar "static" strategy could be used for mobile, where we don't have mouse hover.

[JayPanoz]

So the highlights module has been removed in the develop branch – which means beta.2 release should be imminent.

Nobody complained about this change so we’re assuming this shouldn’t be a breaking change. But be aware that if you were relying on these highlights styles, there won’t be provided any longer.

[JayPanoz]

I guess closing this issue is long overdue. 😅