Add parameters to define the temporal extent

[nigelmegitt]

Split out from #323 - add parameters analogous to the clipBegin and clipEnd attributes of the <audio> element but applicable to the TTML document, whose effect is to define the period of time in the document's timebase for which the document defines some behaviour.

Initial value of the clipBegin parameter would be 0s, which is the current behaviour. There would be no initial value of the clipEnd parameter: if present it would specify a clipping behaviour, or if absent the document contents would define the end time.

The behaviour is as follows:

• The computed value of clipBegin is the specified value if present or 0s if absent.
• When generating ISDs any ISDs ending before the computed value of clipBegin shall be pruned.
• Any remaining ISDs beginning before the computed value of clipBegin shall have their /isd:isd/@begin attribute set to clipBegin.
• If clipEnd is specified, any remaining ISDs beginning after clipEnd shall be pruned.
• If clipEnd is specified, any remaining ISDs ending after clipEnd shall have their /isd:isd/@end attribute set to clipEnd.
  • For the purposes of comparison any ISD with an unspecified end time or an end equivalent to "indefinite" is considered to end after any specified clipEnd time.

Incidentally, it is possible to construct a TTML document with an ISD that has an indefinite end time but it is not possible to express this in an isd:isd document, because the end attribute on the root element is required and <timeExpression> does not permit a value equivalent to the SMIL keyword "indefinite" - I will raise that as a separate issue.

The rationale for not using begin and end attributes for this on the tt:tt element is that the begin attribute would create an offset for the descendant elements' timing, and the tt:tt element is not formally considered to be a time container.

[nigelmegitt]

Raised #484 for the indefinite ending ISD issue.

[nigelmegitt]

As per https://github.com/w3c/ttml2/issues/484#issuecomment-342631941 it would also be possible to allow the "indefinite" keyword on clipEnd.

I would propose that this value be processed as follows:

• if the last ISD generated from the document has a defined end time (as opposed to an indefinite end) then append an additional ISD, being an "empty ISD", i.e. containing no presented content, for example an ISD with no isd:region child, with begin equal to the last ISD's end time, and end being either absent or set to "indefinite" depending on the resolution to #484.

[css-meeting-bot]

The Working Group just discussed Add parameters to define the temporal extent #483.

The full IRC log of that discussion

<nigel> Topic: Add parameters to define the temporal extent #483
<nigel> github: https://github.com/w3c/ttml2/issues/483
<nigel> glenn: If I parse a document and find it only has a caption from 1000 to 1001s then I can
<nigel> .. conclude that anything blank from 0s to 1000s is blank or undefined from the perspective
<nigel> .. of that document.
<nigel> nigel: You can't distinguish between a defined empty presentation behaviour and no
<nigel> .. defined presentation, since there's no way to define an empty presentation for a specific period.
<nigel> .. This is useful for segmentation or live delivery for example.
<nigel> glenn: It could impact the display of backgrounds when showBackground="always"
<nigel> group: [discusses use cases]
<nigel> glenn: I have enough answers to be able to review this now.
<nigel> .. I had a use case for resolving indefinite duration - clipEnd could be used for that.
<nigel> .. That was my intent for defining mediaDuration. I could see that this could be independent
<nigel> .. of having a related media object.
<nigel> nigel: yes
<nigel> glenn: That's why I asked you to consider media duration, at least for the end point.

[nigelmegitt]

Based on @palemieux's comment on #486, and noting that there's a lot of background here, which needs tracing back all the way through #323 and #125 and that we discussed this at some length in last week's face to face meeting, here's a quick summary:

TTML currently has the following problems that this issue is asking for a solution for:

1. A document whose first active time appears to be, say "01:00:00" is ambiguous with respect to the period from "00:00:00" and "01:00:00" - it is not possible to distinguish between the case in which the intention is to define no behaviour in that period, and the case in which the intention is to define "present nothing" in that period.
2. Similarly a document whose last active time appears to be, say "02:00:00" is potentially ambiguous with respect to the period from "02:00:00" to "indefinite".
3. Some documents are a result of processing whose intent is to result in them being active for a specific period of time only, and any content defined outside those times is not intended to be part of their definition. For example documents segmented for fragmented ISOBMFF delivery may not have their begin and end times clipped to the sample period (and this is permitted). It is not currently possible to indicate this intent within the document instance.

Note that there has been discussion in the past about the definition of the root temporal extent. It is not part of this issue to attempt to redefine the document timebase, i.e. move the zero (origin) coordinate in time. However this issue does allow the author to provide some clarity about the root temporal extent as defined.

[palemieux]

@nigelmegitt Thanks for the background. How will these parameters be used in practice, i.e. in actual systems and/or workflows?

[nigelmegitt]

@palemieux I'm going to give a Glenn-ish answer here: that's a matter for profiles and implementations.

For example it could be reasonable to base a future version of EBU-TT Live's document time computation semantics on ttp:clipBegin and ttp:clipEnd rather than/additional to the heuristics present now, or for an implementer of a processor that does temporal segmentation to add the attributes into the output documents so that a subsequent processing step to wrap the documents in ISOBMFF/MP4 files can identify those documents appropriately in a content management and delivery context, like a DASH or HLS manifest. Note that in the latter case there is no conflict with existing standards, since the wrapper mechanism already fully defines the active period for the document.

[palemieux]

I'm going to give a Glenn-ish answer here: that's a matter for profiles and implementations.

Ok, so no changes to document processing then?

[nigelmegitt]

Ok, so no changes to document processing then?

Er, no, that's not what I said - the change is as described in this issue at the top, to truncate any generated ISDs beginning before the defined begin time, and any ending after the defined end time, and probably we should introduce the concept of an empty ISD that, for a defined period, presents no content.

[palemieux]

@nigelmegitt I do not understand why any change to the processing model is necessary. Specifically, I have not seen how generating two identical ISDs around clipBegin serves a purpose. Instead, an application that cares about clipBegin can simply present the ISD from clipBegin to the end of the ISD (or clipEnd, whichever comes first).

[nigelmegitt]

@palemieux No part of my proposal requires generation of two identical ISDs around clipBegin or clipEnd from any one TTML document instance. If you're thinking about post processing of ISDs generated from multiple TTML document instances, that is out of scope of the TTML2 specification right now.

[palemieux]

@nigelmegitt The following implies a modification to the processing model.

When generating ISDs any ISDs ending before the computed value of clipBegin shall be pruned.

[nigelmegitt]

@palemieux yes, it does, but only when clipBegin or clipEnd are present. In their absence, processing is identical to TTML1.

However even in TTML1 the document processing context may apply similar constraints - this is the case already for TTML in ISOBMFF for example. Note that applying these constraints twice at different layers is a no-op, so it is safe.

[palemieux]

However even in TTML1 the document processing context may apply similar constraints

@nigelmegitt Exactly, I do not understand why the processing model needs to change if the application can already do it.

[nigelmegitt]

Exactly, I do not understand why the processing model needs to change if the application can already do it.

@palemieux only applications that use this external context do it. The documents themselves are ambiguous without it.

[palemieux]

@nigelmegitt I still do not understand why the processing model needs to be modified, if external applications already handles in/out points for processing.

[nigelmegitt]

External applications are orthogonal here. The point is that the documents need to express the interval during which they define content, and without this, applications must apply heuristics or externally stated information.

[palemieux]

The point is that the documents need to express the interval during which they define content,

This can be done without modifying the processing model as far as I know.

I do not object to clipBegin and clipEnd.

[nigelmegitt]

It would be nonsensical for the document to express the interval during which it defines content, and then have the processor actually generate ISDs outside that interval.

[palemieux]

It would be nonsensical for the document to express the interval during which it defines content, and then have the processor actually generate ISDs outside that interval.

The processing model generates empty ISDs between subtitle events... that has not been an issue so far, right?

[nigelmegitt]

The processing model generates empty ISDs between subtitle events... that has not been an issue so far, right?

That's not clearly true. There's a reading that it generates gaps during which there are no ISDs. I would like to clarify that the concept of an empty ISD does exist, but that's probably a separate issue.

If a processor generates (even empty) ISDs before or after the time during which some defined behaviour is intended by the document author, then something downstream needs to resolve this. If the context is that there are temporally segmented TTML documents, then that adds complexity.

[palemieux]

then something downstream needs to resolve this.

Yes, it can be resolved using the values of clipBegin and clipEnd, or even better by the media container and/or manifest. In either case, no change to the processing model is necessary.

[nigelmegitt]

@palemieux it's possible for the media container or manifest to do this, but it is not sufficient. I don't really understand what your concern is here with modifying the ISD generation process in this way. Is it a generic concern about making any changes, or a specific concern that it will break something? So far I haven't seen any reason why it will break anything.

[palemieux]

don't really understand what your concern is here with modifying the ISD generation process in this way.

ISDs currently correspond to periods of time where no element is modified, i.e. the document is static. clipBegin and clipEnd breaks this since they introduce ISDs where elements are not modified.

Can you walk me through an actual system where creating these additional ISDs are necessary or even useful? You mention a future version of EBU-TT Live at https://github.com/w3c/ttml2/issues/483#issuecomment-345294497 , but do not explain how the absence of these additional ISDs complicate the current version of EBU-TT Live.

[nigelmegitt]

they introduce ISDs where elements are not modified

They don't introduce any non-empty ISDs - they may modify the begin time of the first ISD or the end time of the last ISD. If you consider that empty ISDs do not exist now, then yes, in principle new empty ISDs could be created, but arguably empty ISDs should already exist. This isn't an academic point by the way.

Can you walk me through an actual system where creating these additional ISDs are necessary or even useful?

I'm not clear that additional ISDs are being created by this change - are you considering the case of empty ISDs?

[palemieux]

I'm not clear that additional ISDs are being created by this change - are you considering the case of empty ISDs?

Yes.

[nigelmegitt]

@palemieux OK, I see. I'll try to explain:

If you are processing multiple TTML documents each representing a temporal segment of some related content, e.g. the same programme, language, purpose etc, and your intent is to generate an output document corresponding to a temporal extent that overlaps with the input documents' temporal extents, then you need to know the temporal extent during which each document defines behaviour.

The way to do this, either conceptually or actually, is to compute the ISDs, apply some precedence rules to define the behaviour* in case there is temporal overlap amongst the input documents, then synthesise an output document from the set of ISDs falling within the desired temporal segment.

In order to identify such temporal overlap by inspecting the ISDs, you need a complete set of ISDs to be generated for each document, including the periods during which no presentation is defined, i.e. empty ISDs may exist, and their presence provides useful information.

* The precedence behaviour is application specific, since we have not yet defined any semantics for TTML sequences.

By the way, the need for empty ISDs to be generated when processing a document exists independently of the clipBegin and clipEnd parameters proposed in this issue. In that sense changing the processing model to produce them is orthogonal to this issue.

[palemieux]

@nigelmegitt I think we are in the weeds, can you walk us through an actual system where requiring processors to create additional ISDs around clipBegin and clipEnd is necessary?

[nigelmegitt]

@palemieux I don't know what you mean about being in the weeds, but here's a concrete example if it helps.

EBU-TT Live section 2.3.1 has to work around the absence of defined begin and end times for documents by using heuristics. In that case the earliest computed begin time and latest computed end times in the document are used as proxies for the defined behaviours (clipBegin and clipEnd would provide a better mechanism). There are worked examples in Annex B.

Those rules mean that an EBU-TT Live document containing something like:

<body begin="01:23:45">
<div>
<p begin="00:00:02">
<span>Nothing for 2 seconds before this text</span>
</p>
</div>
</body>

would have a computed begin time (EBU-TT Live terminology) of 01:23:45, with an implied empty ISD whose duration is 2 seconds until an ISD containing the presented text "Nothing or 2 seconds before this text" begins.

If this example document temporally overlaps a previously received document, and the implementer wishes to use ISDs to resolve the overlaps, then some kind of empty ISD must be created so that the overlap would work.

[skynavga]

@nigelmegitt what is the relationship between

computed begin time (EBU-TT Live terminology)

and the TTML document timeline begin time which is always 0s?

[nigelmegitt]

@skynavga it is the earliest specified begin time or 0s if there is any leaf content element with no specified begin time in the tree all the way up to the root. The actual definition is in the referenced spec at §2.3.1.

[skynavga]

On Tue, Jan 2, 2018 at 3:50 PM, Nigel Megitt notifications@github.com wrote:

@skynavga https://github.com/skynavga it is the earliest specified begin time or 0s if there is any leaf content element with no specified begin time in the tree all the way up to the root. The actual definition is in the referenced spec at §2.3.1.

ok, but keep in mind this is an artificial construct outside the scope of both SMIL and TTML (at present)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/w3c/ttml2/issues/483#issuecomment-354896410, or mute the thread https://github.com/notifications/unsubscribe-auth/AAXCbzLSacAfdbHYEKQ88Kv5F2pGwXXdks5tGrKagaJpZM4QVXnN .

[nigelmegitt]

ok, but keep in mind this is an artificial construct outside the scope of both SMIL and TTML (at present)

@skynavga indeed this issue is open to bring it inside the scope of TTML.

[palemieux]

If this example document temporally overlaps a previously received document, and the implementer wishes to use ISDs to resolve the overlaps, then some kind of empty ISD must be created so that the overlap would work.

@nigelmegitt Can you provide a concrete example of a previously received document that temporally overlaps?

[nigelmegitt]

Can you provide a concrete example of a previously received document that temporally overlaps?

@palemieux Easily:

<body begin="01:23:40" dur="00:00:08">
<div>
<p>
<span>This document temporally overlaps the previous example</span>
</p>
</div>
</body>

[palemieux]

What is the expected visual representation between 01:23:47 and 01:23:48?

[nigelmegitt]

What is the expected visual representation between 01:23:47 and 01:23:48?

In the EBU-TT Live application, the later document takes precedence over the earlier one, so in that case we have for these two documents arriving in order, with both arriving before the real time that corresponds to 01:23:45 and the first example having a later sequence number than the second one:

01:23:40 -> 01:23:45 "This document temporally overlaps the previous example" 01:23:45 -> 01:23:47 [nothing is displayed] 01:23:47 -> undefined "Nothing for 2 seconds before this text"

In this particular application exactly zero or one document is active at any given time. When a document has become deactivated it is never "reactivated".

Note that the WG has agreed to leave such sequence processing out of scope of TTML2 at the current time.

However to enable it to happen elsewhere, observe that it was necessary to know the effective begin time of the first document, which was before it displayed any content, and that the implied empty ISD was effective for two seconds.

[palemieux]

However to enable it to happen elsewhere

What do you mean by elsewhere?

[css-meeting-bot]

The Working Group just discussed Add parameters to define the temporal extent ttml2#483, and agreed to the following resolutions:

• RESOLUTION: Do not apply this change to TTML2, consider using in a future spec version or module.

The full IRC log of that discussion

<nigel> Topic: Add parameters to define the temporal extent ttml2#483
<nigel> github: https://github.com/w3c/ttml2/issues/483
<nigel> Nigel: Reviewing the pull request for this, #486
<nigel> Pierre: If this were metadata I would have no problem with it.
<nigel> Glenn: I'm not convinced that it needs to be a parameter.
<nigel> Nigel: [on whiteboard, explains use/requirement]
<nigel> Glenn: It is possible to generate ISDs that only have metadata in them.
<nigel> .. I'm not convinced that this is needed.
<nigel> Cyril: clipBegin and clipEnd have different definitions in SMIL
<nigel> Nigel: I hadn't noticed that they are in SMIL - I would certainly change the name in that case.
<nigel> .. Thank you for pointing that out.
<nigel> Pierre: I object to changing the processing model.
<nigel> Cyril: Does this proposal generate any changes to timing, like offsets?
<nigel> Nigel: No, it does not modify the relative timings, it just defines those parts of the timeline
<nigel> .. that should be presented.
<nigel> Cyril: That removes one possible problem I was concerned about.
<nigel> Glenn: I agree with Pierre on not changing the processing model.
<nigel> Nigel: Note that ISD generation is conceptual only in TTML1 and no processor needs to
<nigel> .. generate ISDs at all. The serialised format for ISDs is introduced in TTML2.
<nigel> Cyril: It looks like you have a use case Nigel (is that a BBC or an EBU thing?).
<nigel> Pierre: I don't want this to change the processing model.
<nigel> Nigel: What about if its behind a feature?
<nigel> Pierre: Makes no difference.
<nigel> Nigel: To answer the BBC/EBU question: this is not a formal request from EBU. It is something
<nigel> .. that I have observed would be useful for EBU-TT Live and if made available I would
<nigel> .. propose back to EBU that it is used in EBU-TT Live.
<nigel> Cyril: How urgent is this?
<atai2> q+
<nigel> .. Would you be able to defer it?
<nigel> ack atai2
<nigel> Andreas: At the moment there is no communication from EBU about adoption of TTML2.
<nigel> .. We are pretty close to concluding on TTML2 and there is no request, so I'm not sure how
<nigel> .. strongly we should go for that.
<nigel> Nigel: I would rather not put this in if it is metadata - I believe it is parameter data.
<nigel> Glenn: If it is in then we have to have it everywhere in TTML.
<nigel> Cyril: Can we resolve to defer it?
<nigel> Pierre: Or put it in a "live" module?
<nigel> Nigel: [considers this] By omitting this then we retain the much discussed state where the
<nigel> .. root temporal extent is to some degree ambiguous or unclear, whereas including this
<nigel> .. defines it concretely.
<nigel> .. Since we have no consensus on this I'm closing the pull request.
<nigel> RESOLUTION: Do not apply this change to TTML2, consider using in a future spec version or module.