search

shaka-project / shaka-player

JavaScript player library / DASH & HLS client / MSE-EME player
Apache License 2.0
7.11k stars 1.33k forks source link

TTML clockTime at multiple levels incorrectly has an additive effect #3275

Closed david-hm-morgan closed 3 years ago

david-hm-morgan commented 3 years ago

Have you read the FAQ and checked for duplicate open issues? Yes

What version of Shaka Player are you using? 3.1.x

Can you reproduce the issue with our latest release version? No

Can you reproduce the issue with the latest code from master? yes

Are you using the demo app or your own custom app? Yes

If custom app, can you reproduce the issue using our demo app? Yes

What browser and OS are you using? Chrome, Win & Mac

For embedded devices (smart TVs, etc.), what model and firmware version are you using? N/A

What are the manifest and license server URIs?

I cannot share the stream, but here's a snippet of the TTML within exhibiting the issue:

<tt xml:lang="eng"
    xmlns="http://www.w3.org/ns/ttml"
    xmlns:ttm="http://www.w3.org/ns/ttml#metadata"
    xmlns:tts="http://www.w3.org/ns/ttml#style"
    xmlns:smpte="http://www.smpte-ra.org/schemas/2052-1/2010/smpte-tt">
    <head>
        <styling>
            <style />
        </styling>
        <layout>
            <region />
        </layout>
        <metadata>
            <smpte:image imagetype="PNG" encoding="Base64" xml:id="img_0">iVBORw0KGgoAAAANSUhEUgAABfAAAABICAYAAAC5m9TIAAAACXBIWXMAAAsTAAALEwEAmpwYAAAGSklEQVR4nO3dwU4abRSA4WPyX0131ILp1fRP0A16OZWNNWmvphG17Ho77WJCivABMzgwZ+B5Nk1RZw7DRt/5+LgIAACy+9P1AHDCLroeAAAANvmv6wEAAAAAAIB1Aj4AAAAAACQk4AMAAAAAQEICPgAAAAAAJCTgAwAAAABAQgI+AAAAAAAkJOADAAAAAEBCAj4AAAAAACQk4AMAAAAAQEICPgAAAAAAJCTgAwAAAABAQgI+AAAAAAAkJOADAAAAAEBCAj4AALTo98/1xz58Pv4cAABA/wn4AAAAAACQkIAPAHBm2lghfqqrzE/1eQEAAP0k4AMAAAAAQEICPgAAAAAAJCTgAwAAAABAQgI+AAAAAAAkJOADAAAAAEBCAj4AAAAAACQk4AMAAAAAQEICPgAAB/f75/pjHz5X/44GEZ8+Rlxdvv36/UPEbH78uep8z/M84stkv3Nuer5PrxHTx/2Oueu4L7+aXcttr9fiXLc3//5f57Vqcz4AADgXAj4AAJ2ZjCPurstf+z6tQvkxQv4xLKL3cFD++nBQxe19nu+P6fbjRrzvpsOuc93eRMy2HPuY8wEAwCkR8AEA6MS2qLswXKz07nnEr/NcI/49320xfNmumwKrx/4xfd87Bzad6+m1+/kAAOAUCfgAAHSiTtRdfF+TqJ1R3ee6+N7JuN52OnXj+PKx943k28718qv7+QAA4BQJ+AAAdKa0Rc5kXG0lsxx+m0TtJpb3dY/Yvff7ezzP1/e537RC/epy93MtrepfvZ6l4+97LYeDzccvvTvi2PMBAMApEvABAOjE12/lSDt9rFZ0f5++fbxO1M5q0/7us3lEPKw/110m43IcXz3HbF69c2E1pt9dN//g2G3HzzAfAACcIgEfAIBObIvxs3kVfFdXZo8G/Yy627aEKT2fXdvOXF2+/f+uD4D9Mll/d8Gnj82uZZNtbbqYDwAATpGADwBASveFlemibnUTYzXwb/oQ2WWrN0QO9Y6G7PMBAECfCPgAAKRUCvWibnUTY9WmD5Fd9vR6nHc0ZJ8PAAD6RMAHAIAeWd2epq5SRD/EOxqyzwcAAH0i4AMAQM81/RDchWO9oyH7fAAAkJWADwAAAAAACQn4AAAAAACQkIAPAAAAAAAJCfgAANBzHz53PcF22ecDAICsBHwAgDPzPI8YDto/ZttGhRmfXts/T988vbb/+rUp+3wAANAnAj4AAI2UwvohfPq4/tjLr8Oe8xA3N9pWugajQcTsADdR9pF9PgAA6BMBHwCAmIwjpo/1vrcU1g/h6nL9MRG4ugarNxpubyJmk+5mWpZ9PgAA6BMBHwDgzNw/RHyfvn3s6rJaOb0rkI8GEXfX5WM2te2mwWS8vhL+ENv01JFx9fjqNjXDQf2bMKNBdROm7g2bU5wPAAD6QsAHADgzpRXSw0EV9b9+K4f8RVQtxfvn+X6B++66unFw//D25yfj9m4SNFXav/32JiIeckX86WN17ZZn3XQ9I/69fqs/c6hIfoz5fv9cf8yH5QIAcGouuh4AAICd/rR9wNFgfRX+vv6f7I7bpdjaxPM84ssRtmCpe122zbNPWN7nZ0aD6ubCvnv2t/0cVh1yvohWA76/iQAASMsKfACAMzSbV+H9vYG1tJq6bceK9xHldyeUDAfdb60zm0fEw36v4TGuafb5AACgD6w2AQDIr/UV+Mt+TJsH1q/fmm2/UlotXSeUdxVy61yTTbMdawX+srqv4fO82iZo12vX9vY0bc8XYQU+AADnwS+rAAD5HTTgLyzvQ17y9Frvg25LNsXWTed8z7naMhmXr8Wu2boI+BHbX7+m1/MQ+8u3OV+EgA8AwHnwyyoAQH5HCfiH5ANHSczfRAAApGUPfAAAAAAASEjABwAAAACAhAR8AAAAAABISMAHAAAAAICEBHwAAAAAAEhIwAcAAAAAgIQEfAAAAAAASEjABwAAAACAhC66HgAAgJ3+dD0AnDB/EwEAkJYV+AAAAAAAkJCADwAAAAAACQn4AAAAAACQkIAPAAAAAAAJCfgAAAAAAJCQgA8AAAAAAAkJ+AAAAAAAkJCADwAAAAAACQn4AAAAAACQkIAPAAAAAAAJCfgAAAAAAJCQgA8AAAAAAAkJ+AAAAAAAkJCADwAAAAAACf0FX38BNX5TpOYAAAAASUVORK5CYII=        </smpte:image>
        </metadata>
    </head>
    <body>
        <div begin="00:00:03.999" end="00:00:04.109" tts:extent="79% 6%" tts:origin="10% 87%">
            <p smpte:backgroundImage="#img_0" begin="00:00:03.999" end="00:00:04.109" />
        </div>
    </body>
</tt>

I have turned this into a failing UT which can be found at https://github.com/david-hm-morgan/shaka-player/commit/6212f7c3c21d6db5b58f590384a5b5f4861d25bf

What configuration are you using? What is the output of player.getConfiguration()?

drm: {…}, manifest: {…}, streaming: {…}, offline: {…}, abrFactory: ƒ, …}
abr: {enabled: true, defaultBandwidthEstimate: 10000000, switchInterval: 8, bandwidthUpgradeTarget: 0.85, bandwidthDowngradeTarget: 0.95, …}
abrFactory: () => new shaka.abr.SimpleAbrManager()
drm: {retryParameters: {…}, servers: {…}, clearKeys: {…}, advanced: {…}, delayLicenseRequestUntilPlayed: false, …}
manifest: {retryParameters: {…}, availabilityWindowOverride: NaN, disableAudio: false, disableVideo: false, disableText: false, …}
offline: {usePersistentLicense: true, trackSelectionCallback: ƒ, progressCallback: ƒ}
playRangeEnd: Infinity
playRangeStart: 0
preferredAudioChannelCount: 2
preferredAudioLanguage: ""
preferredTextLanguage: ""
preferredTextRole: ""
preferredVariantRole: ""
restrictions: {minWidth: 0, maxWidth: Infinity, minHeight: 0, maxHeight: Infinity, minPixels: 0, …}
streaming: {retryParameters: {…}, rebufferingGoal: 2, bufferingGoal: 10, bufferBehind: 30, failureCallback: ƒ, …}
textDisplayFactory: ƒ textDisplayerFactory()
__proto__: Object

What did you do?

Play stream, enable captions, wait to see the text within a few seconds.

What did you expect to happen? Subtitles appear synchronized to the voice.

What actually happened?

The subtitles are behind and increasing in their offset from expected position.

TheModMaker commented 3 years ago

I think this is working as intended. From my reading of the TTTML spec, section N.2.2 says that media times (which are the default) should be calculated relative to parent elements. This appears to include clock-time expressions in addition to the offset-time expressions.

There is an example in section N.2.3 that shows times being additive. Even though it uses offset-time, the reading of section N.2.2 seems to imply the relative calculation is done for both.

If your reading of the TTML spec is different, please correct me. Timing in TTML is complicated and I'm not an expert.

david-hm-morgan commented 3 years ago

@TheModMaker I am also not an expert at interpreting the TTML spec - I've looked again today to try to find evidence to justify my observation. My concern is that this represents a regression from 3.0.x versions.

TheModMaker commented 3 years ago

Closing as working as intended (unless you find something in the spec). This is probably not a regression, but a fix (since it's what the spec says).