[WR/ARIB] Mixture of text and image

[himorin]

Per: https://github.com/w3c/ttwg/issues/116 Comment 1

ARIB Data WG understands that the draft text of IMSC-1.2 employs Glyph Substitution (GSUB) of OpenType font to add the capability to handle some images in text-only mode. ARIB Data WG thinks that GSUB is originally intended to provide different representation of glyphs (variation) when needed, such as in case of vertical writing or combined forms. ARIB Data WG is afraid that use of GSUB to embed inline graphics in text-only mode may require complicated operation. It may require some additional rules for practical use such as use or not use of Private Use Area (PUA) for a code point of inline graphics. ARIB Data WG would like to point out that vertical writing and Ideographic Variation Selector (IVS) are already in use in ARIB STD-B62 compliant environment. As described in clause 3.2 below, ARIB STD-B62 supports Web Open Font Format (WOFF) thus ARIB STD-B62 compliant environment potentially supports GSUB. ARIB Data WG also understands that use of GUSB assumes the delivery of additional font file(s) with a TTML document. Then it is considered that difference between use of GSUB and simple Gaiji scheme is only manipulation of those glyph images in a TTML processor. Contextual substitution in GSUB is a complicated method, if a target string contains IVS in particular. ARIB Data WG thinks that use of additional fonts with PUA assigned code like Gaiji should be considered as a simpler alternative method for inline graphics. And ARIB Data WG also would like to point out that clear description on use of GSUB for inline graphics in the text of IMSC Recommendation will be helpful for the readers to understand how inline graphics is handled in IMSC-1.2.

[nigelmegitt]

@vlevantovsky any thoughts on this issue?

[vlevantovsky]

Thank you @nigelmegitt, I saw this comment but I am not sure I am fully equipped to offer a solution. In general, I do not like using PUA as a crotch for inserting presentation forms inside text strings, but I do understand their concern here.

Ideally, when you have a font that supports all standardized forms of glyph encoding (outlines, SVG glyphs, bitmaps) - the use of glyph substitution rules to swap one glyph (or a sequence of glyphs) for another would work fine and would be my preferred solution. However, I do understand the concerns related to invoking different rendering environments, e.g. a GSUB substitution that replaces a vector outline glyph with SVG glyph would work perfectly fine if both rendering environments are supported, but this is not always the case.

In general, I think that using PUA for encoding standalone glyph images that are not part of the text string proper (e.g. a standalone image/graphics encoded inline as a character in the end of a sentence or in between words) would be a less-objectionable use of PUA area of Unicode. However, when it comes to using PUA as a crotch to fix the implementation deficiencies (e.g. to avoid complexity associated with chaining contextual substitutions such as GSUB + IVS), I don't think it's a good idea! It will inevitably introduce a wave of incompatibilities between different tools, e.g. those that are used to produce TTML content where implementations are standards compliant. In essence, we would require those tools to adopt special purpose PUA code use instead of using standard Unicode encoding and font-supported glyph substitutions, which would introduce language dependencies (Is it for Japanese only? Is it only for Japanese only when vertical writing is used? Should we only use it for glyph variants / Ruby forms?) and many other problems that are hard to even predict.

[css-meeting-bot]

The Timed Text Working Group just discussed [WR/ARIB] Mixture of text and image w3c/imsc#543, and agreed to the following:

• SUMMARY: TTWG to request ARIB for examples, and consider adding a substitution example to IMSC or TTML.

The full IRC log of that discussion

<nigel> Topic: [WR/ARIB] Mixture of text and image w3c/imsc#543
<nigel> github: https://github.com/w3c/imsc/issues/543
<nigel> Vladimir: Just for clarification, can we clarify exactly what they mean by inline graphics?
<nigel> .. I'm not sure what they really mean.
<nigel> Nigel: I understand it to be the kind of requirement where a company logo is inserted as
<nigel> .. a graphic inline with text.
<nigel> Vladimir: If someone wants to use a PUA code point in a user defined font for this, then
<nigel> .. there is nothing we can do to stop them. I would say that would be a pretty unobjectionable
<nigel> .. use of PUA codes because that's exactly what they're for. It's not going to hinder text
<cyril> this IMSC requirement has an example https://github.com/w3c/tt-reqs/issues/15
<nigel> .. processing or editing or search. I think that is not really a concern.
<nigel> q+
<nigel> .. My concern was about GSUB substitution when you need to select the right form.
<nigel> .. Anything that would be used to avoid doing standard processing for truly text content.
<nigel> .. If they want to simplify things by missing some functionality I would say that's a bad idea.
<cyril> q+
<nigel> .. They say that if GSUB is used before IVS is used then, it sounds like they want to simplify
<nigel> .. by avoiding the need to do it. I think they just want to use PUA to see something they
<nigel> .. want to be displayed. I would say that is a bad idea. It might work in a closed system
<nigel> .. where the Timed Text is authored in the same environment but as soon as you attempt
<nigel> .. to make it something more interoperable then you can't expect everyone to do it the same way.
<nigel> Nigel: I wanted to note that PUA use can impact text processing, for example if a company
<nigel> .. name is being substituted for its logo, then you might reasonably want to do things like
<nigel> .. Text to Speech of the text,
<nigel> .. Searching by text name,
<nigel> .. all before any substitution. If PUA is used that really will break those use cases.
<nigel> .. I think that is why we got to the point of using GSUB as a good idea before.
<nigel> Vladimir: I absolutely agree with this. I don't think that PUA should be used in place of
<nigel> .. a company name, because the implementation will break.
<nigel> q+ Pierre
<nigel> ack n
<nigel> Cyril: I agree with what Vlad said. I talked to my Netflix experts and they are of the same
<nigel> .. opinion that we should avoid PUA as much as possible for all the reasons that were
<nigel> .. explained. I am wondering how ARIB got the notion that we require GSUB, because I
<nigel> .. don't think it is mentioned in the spec. Secondly maybe we should add something but
<nigel> s/but
<nigel> .. to limit the complexity of the implementation. I don't know if we can, for example limit
<Vladimir> q+
<nigel> .. the font functionality required for IMSC. Are there profiles for this? I don't know.
<nigel> ack v
<nigel> Vladimir: Any attempt to do something to simplify implementation to let them off the hook
<nigel> .. of a specific standardised feature, I think is not a good idea, because that feature may
<nigel> .. manifest itself elsewhere that we cannot predict.
<nigel> .. If normal text comes in and implementations drop a standardised case then
<nigel> .. preprocessing would be needed. The short answer: I don't think it's a good idea to
<nigel> .. simplify implementations if it goes against the standardised feature set.
<nigel> ack Pier
<nigel> Pierre: I think what we should do is get actual samples. It just occurred to me that there
<nigel> .. are no examples of what they are trying to do. We should try to see how this works
<nigel> .. in practice. We had a long thread on what we wanted to do with GSUB and we should
<nigel> .. try it and assess how well supported and how easy it is.
<nigel> .. We're at the point where we need to try it before we come up with a solution.
<nigel> .. I would actually go back to ARIB and request a sample.
<nigel> Nigel: That's a really good idea
<nigel> Cyril: Are there tools that allow us to easily create fonts with a GSUB substitution?
<nigel> Vladimir: Any font tool - most of them allow substitutions. You write your own rules as
<nigel> .. a code entry, to substitute a sequence of glyphs. You don't know what those glyphs
<nigel> .. represent. You just set a rule. Any sequence of input can be substituted. For example
<nigel> .. a company name substituted by a logo is perfectly possible.
<nigel> .. For example the Zapfino font, on most Macs I think, has a substitution entry that
<nigel> .. substitutes the name of the font for the ligature. They do it just to showcase it.
<nigel> .. You can substitute a ligature or anything else.
<nigel> Nigel: That's an input sequence of code points?
<nigel> Vladimir: The input is a sequence of unicode text points.
<nigel> .. Then map those to glyph ids.
<nigel> .. Then most of the time the substitution rules apply to those glyph ids
<nigel> .. Then you have character codes, and depending on location and many other rules, the
<nigel> .. base glyph can be substituted by something else.
<nigel> .. For example in Arabic, a positional variant; for Japanese, a variation sequence definition.
<nigel> .. If you have a ligature for example for a sequence of glyphs, that is applied to the glyph
<atai> q+
<nigel> .. id sequence mapped from the character codes.
<nigel> .. You end up as part of this process as one code point entry mapped to a glyph that is
<pal> q-
<nigel> .. one of a number of possibilities.
<nigel> ack c
<nigel> ack at
<nigel> Andreas: A question re GSUB and PUA. Regarding the concerns that Nigel mentioned
<nigel> .. for example using text for a screen reader, where is the difference between GSUB and
<nigel> .. the use of PUAs? Both are not very accessible.
<nigel> Vladimir: It's exactly opposite. Your accessibility is defined by the code point sequence.
<nigel> .. Then your Unicode sequence does not change and is used by the screen reader.
<nigel> .. The font level modifications will only affect visible display, not the content itself.
<nigel> .. That is why this is probably the only accessible way of doing things.
<atai> q+
<nigel> .. If you move visualisation decisions upstream and simply use a PUA code point to map
<nigel> .. to a particular glyph, then you break accessibility, because now your screen reader has
<nigel> .. no idea what that is.
<nigel> ack at
<nigel> Andreas: And there is no requirement that the mapping of the glyphs that people will read
<nigel> .. will go with what is specified by the code points.
<nigel> Vladimir: Exactly, which is why PUA should be avoided unless there is something that has
<nigel> .. no meaning for somebody who cannot see the text.
<cyril> Zapfino example: data:text/html,<div%20style="font:%2048px%20Zapfino">Zapfino<br>Zapfin%20o
<nigel> .. If you have "company name, logo" where logo is a PUA then that's fine if the screen reader
<nigel> .. ignores the PUA but if the company name is omitted then it will not be accessible.
<nigel> Nigel: This reminds me of presentation-scheme based fallback options, and I think we
<nigel> .. should avoid those if we possibly can.
<nigel> Vladimir: Yes exactly, and that is the basis of the Unicode choice to let font engines
<nigel> .. do substitutions where needed so that they only affect visual presentation.
<nigel> Cyril: I asked my font expert if there is a limit to the length of the substitution, and he
<nigel> .. told me it can be very long, like 30 glyphs. Is there a limit?
<nigel> Vladimir: I don't think so, only practical limitations.
<nigel> .. Substitution tables can define a chain of substitutions and it is only limited by
<nigel> .. complexity and how far a font designer wants to go.
<nigel> q?
<nigel> Cyril: Vladimir you were asking for an example. Earlier on IRC I posted a link to one of the
<nigel> .. requirements that we have. Nigel showed an example of the Twitter logo inline with the
<nigel> .. text.
<nigel> Nigel: Thanks for digging that out!
<nigel> Cyril: It's w3c/tt-reqs#15
<nigel> Pierre: That's the issue that led to the current situation in IMSC and TTML.
<nigel> .. It would be good to get input from ARIB with sample text and corresponding render.
<nigel> Vladimir: Yes, for example if someone wants to define the logo as a PUA code in additoin
<nigel> .. to the name Twitter, then that would be fine.
<nigel> .. But if you drop the name and only use the PUA for the logo, it breaks accessibility.
<nigel> .. Better to do it as a font substitution, for visual presentation.
<nigel> .. As far as content sequences are concerned the name Twitter is still there.
<nigel> Cyril: Also graceful degradation, in case the font engine doesn't support substitution.
<nigel> Vladimir: I agree. Any time substitution fails you see the original unsubstituted text.
<nigel> .. For company names that's fine. For Devanagari almost everything is a substitution, so
<nigel> .. the presentation would fail.
<nigel> Cyril: I'm trying to get back to the ARIB issue and understand what exactly they wanted.
<nigel> .. They end by saying to consider that PUA is a simple implementation and a clear indication
<nigel> .. on the use of GSUB would be helpful.
<nigel> .. On the first point I think we disagree with them. We don't want to recommend it.
<nigel> Pierre: I can't even conclude that without seeing what they want to do and making sure
<nigel> .. that we can do it.
<Vladimir> q+
<nigel> Cyril: I agree that would be useful. I wonder if we should say that PUA is not recommended.
<nigel> Pierre: Imagine they come back with a PUA example where we can't give a better alternative.
<nigel> Cyril: You would want to say at this stage we cannot ...
<nigel> Pierre: I would like to get a solid example.
<nigel> Cyril: Yes, okay that's good.
<nigel> Pierre: If they cannot produce an example that also informs us a lot.
<nigel> Vladimir: 2 final comments. One on what was just discussed. I don't think we can do anything
<nigel> .. to stop them using PUA codes. If someone decides to use it we cannot prevent it.
<nigel> .. On the substitution side, trying to define something in the TTML spec, all we can say
<nigel> .. is we expect font engines to be conformant with the OFF standard.
<nigel> .. If they support the standard then that's not a concern.
<nigel> Cyril: I don't think we want to explain how substitution works in general, but maybe
<nigel> .. an example of how to use substitution to explain how it can be used to produce
<nigel> .. inline graphics could be useful in TTML.
<nigel> Vladimir: That would be fine [assuming that the spec is stuck to]
<nigel> Cyril: I agree.
<nigel> SUMMARY: TTWG to request ARIB for examples, and consider adding a substitution example to IMSC or TTML.
<nigel> Nigel: I think I heard no proposals for substantive language about support for particular
<nigel> .. features.
<nigel> Cyril: I think Pierre [who left a moment ago] was saying we should wait for examples first.
<Vladimir> just an illustration to my previously used example: https://www.myfonts.com/fonts/linotype/zapfino-extra/
<nigel> Cyril: The action is to request this from ARIB as part of a general response?
<Vladimir> the whole name is substituted with the glyph that represents the font name
<nigel> Nigel: I would prefer to wait until we have covered the other ARIB issues but if it is going to
<nigel> .. be many weeks then I would prefer to do it sooner.
<nigel> Cyril: Yes that makes sense.
<cyril> data:text/html,<div%20style="font:%2048px%20Zapfino">Zapfino<br>Zapfin%20o