Updating cue time attributes can generate unexpected events

[rjksmith]

Updating the startTime or endTime attributes of a cue (TextTrackCue or DataCue) can generate unexpected enter and exit events depending on the sequence in which the steps happen.

For example, if the cue endTime is updated to a later time just before expiry, it is possible to generate an exit event followed by an enter event as follows:

1. Identify an active cue to be updated, i.e. prior to its end time;
2. The cue end time passes;
3. Update the cue endTime to a later time.

Running the time marches on algorithm during this process producing differing results depending on where it occurs in the sequence of steps:

• Time marches on algorithm runs between steps 1 and 2: no events are generated as the cue is still active when it is updated;
• Time marches on algorithm runs between steps 2 and 3: an exit event is generated as the cue expires before it is updated and an enter event will be generated when the algorithm next runs.

It should be noted that both these outcomes are correct, but generating multiple enter and exit events for a single cue may be unexpected behaviour and users should be made aware of this possibility in order to design their code properly.

A similar issue can occur if the startTime of an inactive cue is updated to a later time just before it is due to start.

[nigelmegitt]

By what mechanism is it possible to update a cue endTime in the middle of executing time marches on?

I ask because I seem to recall that javascript runs single-threaded and I don't expect client javascript code to get any chance to run in the middle of the algorithm. Is there a way for them to run on separate threads without appropriate resource locks being in place, for example?

[rjksmith]

To clarify, I mean that the 'time marches on' algorithm can run between steps 1-3, as it's typically triggered by a timer and interrupts the application thread. I do not mean that 'time marches on' can be interrupted.

[tidoust]

By what mechanism is it possible to update a cue endTime in the middle of executing time marches on?

Nothing ties the "time marches on" algorithm to the event loop and I suspect all browsers run it in a separate thread in practice.

Updating endTime on a cue triggers a new run of the "time marches on" algorithm according to the spec.

The spec does not say much about what needs to happen if the algorithm is already running when this happens. Is it up to implementations? Will the update be immediately taken into account?

The statement "the user agent must wait for the steps to complete, and then must immediately rerun the steps" is probably meant to apply to that situation as well, meaning that the algorithm will re-run. But first run can probably still create the unexpected exit event.

[nigelmegitt]

Thanks for the additional explanations @rjksmith and @tidoust . The first line in the issue says:

... can generate unexpected enter and exit events

The trouble I have with this statement is that I think the events are entirely expected in the circumstances described, and I don't see how it can be any other way. This issue describes a race condition between a cue ending (or starting) and its end (or start) time being updated. As the time when the update is applied tends towards the time being updated (or the new time post-update), then this race condition becomes less deterministic, with actual observed behaviour likely to depend on factors beyond the control of a specification.

If I've read this correctly @rjksmith you're suggesting that not everybody might realise that this issue could occur, and some folk might occasionally get caught out by it, so you want to call attention to it somewhere? What would you say?

Practically speaking, I have been unable to think of any steps that could be taken either in user agent implementations or in user code to mitigate the issue.

If the issue is causing a regular problem for a particular user, then presumably they have an application that sits right at the "leading edge" and is often updating cue times very close to the times being updated, perhaps because they have a very low latency system that runs "just in time". I'm not sure that advising such a user to increase the latency of their system would be what they would want to hear!

[chrisn]

This is something we should consider in the DataCue spec, as cues could be updated either through application script or by being updated by the UA, e.g., for in-band events.

It's not clear to me whether this issue should result in a proposed change to HTML (e.g., to add an informative note), or whether it doesn't need calling out in the spec as it's expected behaviour.

[rjksmith]

I agree. A key issue is whether a cue is generated by the app or user agent, and how to correctly handle and discriminate between those cases.

I'm not proposing any changes to HTML at this stage, but am keen to ensure any changes we do propose address the proper issues.

The expected behaviour for a cue is to raise one enter followed by one exit, which is always the case if the start time is reached and attributes remain unchanged. Updating a cue time can alter this behaviour, so we should explore the possible effects and mitigatation mechanisms to determine whether particular use cases are already supported by the existing API and identify new requirements, without introducing spurious ones.

[nigelmegitt]

The expected behaviour for a cue is to raise one enter followed by one exit

I think there are some unstated assumptions behind this statement, which are worth being explicit about. I think there is only one specific case in which that is the expected behaviour, which is when the associated media is played through exactly once with no pause or seek events.

In any other case there will very likely be some other number of enter and exit events, generally, from zero if the media is not played at all, to any higher number if the media is replayed, seeked etc. Of course cues can be added during playback already. The time marches on algorithm is designed to deal with all those cases.

The higher level question is how well that algorithm deals with all those scenarios when the cue times themselves are also being modified. I agree there's no specific problem highlighted so far that would motivate a change to HTML, and that in proposing any changes to the Text Track Cue interface we should be careful not to introduce any such problems, or if they are unavoidable, propose a solution.

[rjksmith]

The events are unexpected because there are circumstances in which a user cannot know whether changing a cue time attribute will generate either one or two pairs of enter and exit during playback to the end of media.

[nigelmegitt]

Generally users don't change cue time attributes, but content providers might. Content providers can, in general, never know how many enter and exit events a cue will generate, unless they lock down the playback options or do something unusual like having an exit handler remove the cue that fired it.

[rjksmith]

1. I mean any user of the API - including applications, user agents and content providers.

2. I mean that updating a cue time attribute during playback may or may not generate spurious events as a result of that action.

[nigelmegitt]

I'm thinking about this thread and realising that I feel like I'm being a bit, I don't know, snipey, picking up on seemingly minor details, so a) I probably am and b) I'm sorry about that @rjksmith .

It isn't at all clear to me that there's actually anything wrong here. What is a "spurious" event seems to be in the eye of the beholder.

If you imagine yourself in the position of the user agent processing all the incoming events, including those that update cue times, and those that have been generated during the course of handling events, you're just doing your job, everything's fine. None of the events is spurious, they're all just events you're required to fire. Keep on rolling.

If you imagine yourself in the position of the user observing some presentational artefact of a cue, there may very occasionally be some visible strange behaviour for cues that might be subject to late changes.

Looking back at the original issue text:

It should be noted that both these outcomes are correct, but generating multiple enter and exit events for a single cue may be unexpected behaviour and users should be made aware of this possibility in order to design their code properly.

• Yes, the outcomes are correct.
• Are the events unexpected behaviour? It would be good to have some data rather than the hypothetical. This seems heavily predicated on the design of the implementation, and what it is intended to achieve.
• Can a user make any code changes to vary the behaviour more to their liking? It's hard to see what could be done given we haven't got an example of some preferred behaviour.