Add embedded image support to IMSC Text Profile

[nigelmegitt]

This issue relates to to w3c/imsc-vnext-reqs#4 for consideration by TTWG in 2019.

This is not about adding embedded image support to IMSC Image Profile - for that see #16.

---

Is your feature request related to a problem? Please describe.

Use case: We need to include small inline images alongside text, in SVG and possibly other image formats. Ideally those inline images would be embedded content.

Describe the solution you'd like

In IMSC Text Profile we would like to be able to include an inline image to be laid out as a grapheme cluster inline with text, whose size would be adjusted using tts:fontSize and whose image content would be embedded in the file either directly (locally in the file) or as a resource defined in the head and referenced from the content, or as a remote resource.

This is likely to need some modification to TTML2 to support the sizing and referencing aspects, since it is asking for an image to be used like a glyph.

Describe alternatives you've considered

• Using an image profile IMSC document would allow pre-rendering but not allow for accessible rendering of the text.
• Mapping to an HTML i element with a relevant CSS class and a stylesheet such as provided by, e.g. fontawesome allows for automated insertion of content in place of the i element. This is still an image but is easier to work in an inline, font size based way.
• Using a special font such as Social Icons which places non-approved Unicode symbols into the private use area (E000-F8FF) allows for insertion of selected images but necessarily in a non-standard way.

State if you intend for this requirement to be met by a particular specification

Likely to need changes in both TTML and IMSC.

Does this requirement represent a change in scope

Could be considered a small increase in scope of IMSC, but more generally, no, within current scope.

Additional context

See https://github.com/w3c/tt-reqs/issues/15#issuecomment-454075897 for an example image.

[nigelmegitt]

Adding another facet to this requirement:

I recall forwarding to the WG, either in conversation or in an issue (which I now cannot find) a requirement requested from a Japanese contributor to me in person at the HbbTV Symposium in Rome, 17th and 18th October 2017. This requirement was to permit inline presentation of images in text subtitle documents, with the use case being to represent major brand logos in subtitle text.

Coincidentally a colleague at the BBC told me today that, in using TTML to represent on-screen captions there is a need to put in SVG images inline with text. This is not for translations or hard of hearing subtitles, but for styled narrative texts or quotes provided as data for player-based rendering rather than burned into the video image as video pixels. Such captions are typically "forced" and interleaved with non-forced hard of hearing or translation subtitles. The SVG images are for example a Twitter logo, if a quote is being attributed to a tweet. It may be that where an SVG image file is not available a PNG or other format could be used instead.

[updated: completed template moved to first comment in issue]

[nigelmegitt]

Example image:

The text at the top right includes an icon, in this case a twitter icon.

[nigelmegitt]

I've found the place this was discussed previously: w3c/imsc-vnext-reqs#4

[nigelmegitt]

Business test for this:

When a symbol that has no Unicode character (e.g. gaijin) needs to be presented inline with text, content authors can insert the image easily, distributors can distribute it with minimal changes to distribution infrastructure, and players present the images consistently with authoring expectations.

[edited to respond to @skynavga 's comments]

[palemieux]

@nigelmegitt https://github.com/w3c/imsc/issues/82 was asking for embedded images for the purpose of transmitting complete timed text programs. It sounds like you are thinking about inline images to supplement text-based timed text programs.

[palemieux]

@nigelmegitt How would altText be carried/conveyed in the case of a inline image to be laid out as a grapheme cluster inline with text.

[nigelmegitt]

As @palemieux pointed out, we need to separate the sending of images for Image profile vs supplementing text in Text profile with graphics not available as Unicode text. I'll take the action to clean this up.

[nigelmegitt]

The easiest way to clean this up was to make this into an issue about embedding images in IMSC Text profile documents and open a new issue (#16) about embedding images in IMSC Image profile documents. Tidy-up now completed.

[nigelmegitt]

@palemieux your question in https://github.com/w3c/tt-reqs/issues/15#issuecomment-455230792 is a good one.

I'd add to the requirement like this:

• When an inline image is presented, a text alternative should be available.
• Screen readers should be able to read the alternative text.
• Optionally, for presentation processors unable to present the image, they should present the alternative text instead.

In the context of the example with the Twitter icon, a screen reader may be expected to read the word "Twitter", say, in place of the image.

[nigelmegitt]

@cconcolato asked me if the image support needs to include colour. It does - the monochrome Twitter example is not the general case. For example some company logos that may be used in this way need to be presented in colour to work (especially the badly designed ones?!).

[cconcolato]

To me, there are 2 possible approaches:

• using images
• using fonts

I see several points to discuss regarding the image approach:

• Accessibility (see @palemieux comment above).

TTML2 has an example of how altText metadata items can be used for that:

<p>
  <image src="#talking-image">
    <ttm:item name="altText" xml:lang="en">Talking</ttm:item>
    <ttm:item name="altText" xml:lang="fr">Parlant</ttm:item>
  </image>
</p>

So there does not seem to be a need for a technical change for TTML2. I would think though that having a mention of that approach in the image section would be very useful.

• Sizing images with fontSize:

@nigelmegitt Could you clarify the need for this requirement? I can see that if your UI lets the user increase/decrease the font size you would want the Twitter logo to be resized. Any other use case? IIUC, the sizing of an image is done with tts:extent, which can specify a <length> including using the em unit. Would this solve this sizing issue?

• Use of images in IMSC Text Profile

@nigelmegitt I don't understand the second sentence in your initial description:

This is not about adding embedded image support to IMSC Text Profile - for that see #16.

16 is about IMSC Image Profile. "Image" not "Text". Is it a typo in your issue description?

@nigelmegitt also the following part in your issue description says:

Using an image profile IMSC document would allow pre-rendering but not allow for accessible rendering of the text.

Can you clarify why you think this would not be possible? Is it due to IMSC Image profile restrictions on TTML2 images?

In any case, this approach would require changing the restrictions in the IMSC Text Profile, and we'd have to be careful because we don't want to require complex image processing in the text profile.

Now about the second option, using fonts. My understanding is that with this approach, you would need to author the content with:

• a special font that maps the sequence of 'T', 'w', 'i', 't', 't', 'e', 'r' (or any other sequence, like 'TwitterLogo', possibly preceeded with '[' and followed by ']') into the 'Twitter' logo glyph
• and use that font only for words that you want to substitute (i.e. put it in a <span tts:fontFamily="TwitterLogoFont">) and not for the others.

It is by definition accessible and sizing also works fine using tts:fontSize. This also would require change in IMSC, and we'd have to be careful too about the combination of features (font, embedding, WOFF, ...) that we want to have.

Also it should be noted that in both approaches (images or fonts), both raster or vector graphics images can be used.

[skynavga]

Re: https://github.com/w3c/tt-reqs/issues/15#issuecomment-456515719 about sizing images with tts:fontSize, this is incompatible with both CSS and TTML2 style semantics, so this is not going to happen.

• Sizing images with fontSize:

  @nigelmegitt Could you clarify the need for this requirement? I can see that if your UI lets the user increase/decrease the font size you would want the Twitter logo to be resized. Any other use case? IIUC, the sizing of an image is done with tts:extent, which can specify a including using the em unit. Would this solve this sizing issue?

The only way to do this is to put the image into a font (as spline curves), in which case font size already applies.

[nigelmegitt]

@cconcolato re scaling images with font size, that's the requirement, with the use case as you identified. I haven't tried specifying a height with em units but it's an interesting idea.

From the accessibility point of view, another way to do this would be to make the content text by default and add a "replace with SVG" attribute to the span containing the text. For example:

`Great Brand and have the semantics of presentation of that replacement image set the height to 1em.

I think that kind of use of a font, where a sequence of letters is automatically replaced, is probably the hardest authoring and implementation option of all of them. It means that document authors will really not be sure what is likely to be displayed.

@skynavga I'd like to keep investigating, I'm not really convinced yet that it is incompatible with CSS.

The bit about it not being about Text Profile looks like a typo - I will update it.

[nigelmegitt]

I've updated the bit about Text Profile to say Image Profile @cconcolato .

[skynavga]

On Thu, Jan 24, 2019 at 8:11 AM Nigel Megitt notifications@github.com wrote:

@skynavga https://github.com/skynavga I'd like to keep investigating, I'm not really convinced yet that it is incompatible with CSS.

I am (completely convinced). Font size cannot apply to an object that has no definition of EM square. We have tts:extent and tts:{bpd,ipd} for sizing inline images and their generated areas.

[nigelmegitt]

@skynavga CSS has separated out values and units, so https://www.w3.org/TR/css-values/#font-relative-lengths for example shows the basis of em units as being the font size of the element. Any length can be specified in em units even if it is not text.

By way of a proof, here's a very simple example that demonstrates sizing of an SVG by font size: https://codepen.io/nigelmegitt/pen/KJgmvp

[skynavga]

There is no problem with respect to using em units with any of the size related styles that apply to images. That is because em is resolved into pixels by making reference to the current font size. So in this (indirect) respect, yes, tts:fontSize may be employed for resolving em units.

So, I guess you may be asking how em can be used by an element for which tts:fontSize is not applied, strictly speaking?

[nigelmegitt]

So, I guess you may be asking how em can be used by an element for which tts:fontSize is not applied, strictly speaking?

@skynavga I don't think I am asking that, no, because the request here is for an inline image to be sized with the text it is inline with, which by definition therefore does have a font size. I expect that using the parent element's font size is acceptable too, i.e. that there would be no need to try to "grab" the font size of a sibling element somehow.

[skynavga]

@nigelmegitt keep in mind that an image can appear as a child of div, p, and span. In the first two of these cases, it has block formatting semantics; in the third case, it has inline semantics.

If tts:fontSize is to be used to resolve em unit usage with respect to, say, tts:extent, then it would have be resolved with respect to the computed value of tts:fontSize on any of these three elements. It is probably worth clarifying this case where one might possibly argue that tts:fontSize is being applied to image.

[nigelmegitt]

Indeed, and I see that although tts:fontSize only applies to span it has a computed value on ancestor elements including p and div which is used in font size computation.

[css-meeting-bot]

The Timed Text Working Group just discussed Add embedded image support to IMSC Text Profile tt-reqs#15, and agreed to the following:

• SUMMARY: Solution options available, need more thought and investigation, overall requirement to find some way to present images like glyphs are presented, in an accessible way, is accepted.

The full IRC log of that discussion

<nigel> Topic: Add embedded image support to IMSC Text Profile tt-reqs#15
<nigel> github: https://github.com/w3c/tt-reqs/issues/15
<nigel> Pierre: Remind me why fonts wouldn't solve this?
<nigel> Nigel: It's not interoperable and it is not accessible.
<nigel> Glenn: You can use private use area easily.
<nigel> Nigel: You have to manage authoring and distribution fonts carefully.
<nigel> .. Also there's no text alternative.
<nigel> Cyril: You can use glyph substitution so there is a text alternative.
<nigel> Glenn: It's implementation dependent.
<nigel> .. You can embed the font in the document to ensure the private use area usage is consistent.
<nigel> Nigel: If we can't use glyph substitution then you have the accessibility issue. There's no text alternative.
<nigel> Pierre: You can put altText on the span containing it.
<nigel> Cyril: If you have text in English how do you select altText in English.
<nigel> Glenn: You can use it you can do whatever you want with it.
<nigel> Cyril: Yes so you can't use it because there's no semantic defined.
<nigel> Cyril: Is glyph substitution implementation widespread?
<nigel> Glenn: g subtable
<nigel> Vlad: Yes it is universally implemented and supported in all browsers.
<nigel> Glenn: So native TTML renderers or translators like imsc.js it is not supported unless you map the font
<nigel> .. into the browser world and pass along the string, maybe it could be supported that way.
<nigel> Vlad: Just to give me a bit better understanding of the subject, what do you think would stop this?
<nigel> Glenn: For gaijin characters, and emerging emoji.
<nigel> Nigel: And things like company logos.
<nigel> Glenn: The idea is to use a downloaded or embedded font and use PUA code to map to a glyph.
<nigel> Nigel: Or glyph substitute a set of characters like "twitter" to the twitter icon.
<nigel> Vlad: That works - for example the word Zapfino in the Zapfino font gets replaced by a logo for the font.
<nigel> Pierre: If you go to a text alternative as a fallback there's a bigger issue that it may require a different layout.
<nigel> .. The entire idea of having an image fallback be unstyled simple text doesn't work very well. How do you
<nigel> .. specify the style (color, italics) of that fallback. That's why in IMSC you have a separate file for images.
<nigel> .. The text equivalent of image is not always what is written in the image.
<nigel> Vlad: Remind you that fonts can have glyphs defined in SVG and that glyph can be selected when conditions
<cyril> q+
<nigel> .. are right. It will still be text for all intents and purposes.
<nigel> Pierre: When you substitute long text with one emoji that might affect the layout significantly.
<nigel> Vlad: I agree
<nigel> Pierre: that is a significant issue here.
<nigel> Vlad: Rendering differences are not as bad as they used to be. Using a particular font is probably the only
<nigel> .. way to make sure the rendering is right. Assuming that some font will be right is wishful thinking.
<nigel> Glenn: We don't define line breaking or space distribution semantics so implementation dependent.
<nigel> ack at
<nigel> ack c
<nigel> Cyril: Pierre raised an interesting point, the layout. I agree replacing a small image with 10 or 20 characters
<nigel> .. is a big change. Focusing on the requirement, is there any objection to the requirement being agreed?
<nigel> Glenn: I have a problem with the requirement because it is written in terms of images not text.
<nigel> Pierre: What about recasting it in terms of glyphs?
<nigel> Cyril: Using inline images with alt text does the trick, right
<nigel> s/t/t?
<nigel> Glenn: That's the traditional way with gaijin.
<nigel> .. We have a terminology issue with Text and Image profiles each being constrained.
<nigel> Pierre: The entire discussion so far has been really about custom characters, right?
<nigel> Nigel: Yes, I don't think we want to support generic pre-rendered text in an image into text profile.
<nigel> .. However the change doesn't make a substantive difference.
<nigel> Cyril: You can put any image into the font though, right?
<nigel> Glenn: Correct you could.
<nigel> Cyril: You want to say that if you don't make fonts available the content of the file should be human readable and carry the content?
<nigel> Pierre: I don't agree with that
<nigel> Glenn: I don't see that as a requirement.
<nigel> Cyril: I'm not sure of the requirement, we're talking about solutions.
<nigel> Pierre: This is for visual elements as text.
<nigel> Nigel: The only requirement that ties it to text is to size the image using the font size.
<nigel> Glenn: Also font kerning, spacing, scaling etc are all affected by laying out as text.
<nigel> Nigel: Recasting, the request back seems to be to support custom font embedding in IMSC Text profile.
<nigel> Pierre: That is how it is done in ARIB-TT.
<nigel> Cyril: I don't think it is the same thing.
<nigel> .. This is not the gaijin character use case. That gaijin is readable, but the private use area character is
<nigel> .. not readable.
<Vlad> q+
<nigel> Pierre: It can be a cartoon.
<nigel> Cyril: It is not accessible, as Nigel explained earlier, if you don't process the font the text is still readable
<nigel> .. by a screen reader.
<nigel> ack v
<nigel> Vlad: I've been listening to the discussion. As far as the requirement is concerned there are three
<nigel> .. requirements that should be connected.
<nigel> .. 1. Support custom fonts by embedding
<nigel> .. 2. Support of inline images as part of text (not how it can be done, but that seems to be the requirement)
<nigel> .. 3. (blanking out on that!)
<nigel> .. Having font embedding functionality is a way to support inline glyphs.
<atai1> q+
<nigel> Nigel: Also we need a screen reader to do something sensible here.
<nigel> Glenn: The two options are potentially orthogonal solutions to this problem.
<nigel> Vlad: Not necessarily.
<nigel> Glenn: The embedded font is more consistent with treating this as text and is more compatible with the
<nigel> .. text profile which does not include image, but as has been pointed out it might not satisfy some
<nigel> .. screen readers but I think they are out of scope.
<nigel> Nigel: No way are screen readers out of scope.
<nigel> Vlad: Accessibility was my third requirement.
<cyril> q+
<nigel> .. All those requirements are valid concerns and there may be one solution that satisfies them. If we can
<nigel> .. agree on the requirements that would guide us and many other implementers on the solution.
<nigel> ack atai1
<nigel> Andreas: Can we bring this to a close (time)?
<nigel> ack c
<nigel> Cyril: I agree with Vlad's phrasing and support all three of those requirements.
<nigel> .. We could use this heavily at Netflix.
<nigel> Glenn: I do not agree that we have separate requirements for embedded glyph and image support in text.
<nigel> .. They are two different scenarios for solving one requirement, which is to support the ability to display
<nigel> .. as text some form of an image either as a glyph or an inline image characters for which there is no
<nigel> .. standardised visual representation. There's probably something on accessibility too if you cannot
<nigel> .. display the rendering. I view Vlad's first two requirements as two different solutions to the underlying same requirement.
<nigel> Nigel: +1
<nigel> SUMMARY: Solution options available, need more thought and investigation, overall requirement to find some way to present images like glyphs are presented, in an accessible way, is accepted.
<gkatsev> +q
<nigel> ack at
<nigel> ack gk
<nigel> s/+q/

[nigelmegitt]

See also the discussion at w3c/csswg-drafts#4116.