fMP4 containing multiple moof cannot be played back properly in progressive mode (v1.4.2 regression)

otya128 commented 1 year ago

What version of Hls.js are you using?

v1.4.2-

What browser (including version) are you using?

Firefox 115.0.2 (x86_64), Google Chrone 115.0.5790.110 (x86_64)

What OS (including version) are you using?

Windows 10

Test stream

https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8
https://stream.mux.com/v69RSHhFelSm4701snP22dYz2jICy4E4FUyk02rW4gxRM.m3u8

Configuration

{
    "progressive": true
}

Additional player setup steps

No response

Checklist

[X] The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
[X] The issue occurs in the stable client (latest release) on https://hlsjs.video-dev.org/demo and not just on my page
[X] The issue occurs in the latest client (main branch) on https://hlsjs-dev.video-dev.org/demo and not just on my page
[X] The stream has correct Access-Control-Allow-Origin headers (CORS)
[X] There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

Play a fMP4 HLS contains multiple moof+mdat in one segment longer than 1 second.
Enable progressive mode.

Expected behaviour

Playback properly.

Introduced in #5471 according to bisect.

$ git bisect log
# bad: [fc5e295c3b045d750dc88edfe71249368d58211c] Allow live level loading to recover from net::ERR_NETWORK_IO_SUSPENDED errors (#5473)
# good: [b0a72ff5ddcb1ec03137947033837e416c6eb982] Merge pull request #5454 from video-dev/renovate/microsoft-api-documenter-7.x
git bisect start 'v1.4.2' 'v1.4.1'
# good: [8fe26396d05505727186c8bfedb1dd1aecb65ad9] Merge pull request #5467 from video-dev/renovate/rollup-3.x
git bisect good 8fe26396d05505727186c8bfedb1dd1aecb65ad9
# bad: [9ed55f123cca70ed25c457bd573a0d44083260d2] Fix AV desync regression in v1.4.0 when mp4 audio track timestamps start before video track timestamps (#5471)
git bisect bad 9ed55f123cca70ed25c457bd573a0d44083260d2
# good: [8f11b06f683657b45713f5476da3456f7a74e102] chore(deps): update dependency eventemitter3 to v5.0.1
git bisect good 8f11b06f683657b45713f5476da3456f7a74e102
# good: [ef718d2302a8aef7271d7c22787348c66a6db46e] fix: partial audiovideo fragments not being treated as partial (#5460)
git bisect good ef718d2302a8aef7271d7c22787348c66a6db46e
# first bad commit: [9ed55f123cca70ed25c457bd573a0d44083260d2] Fix AV desync regression in v1.4.0 when mp4 audio track timestamps start before video track timestamps (#5471)

What actually happened?

Playback stuck periodically.

Console output

[log] > [stream-controller]: WAITING_LEVEL->IDLE
base-stream-controller.ts:727 [log] > [stream-controller]: Loading fragment initSegment cc: 0 of [1-76] level: 17, target: 0
base-stream-controller.ts:1761 [log] > [stream-controller]: IDLE->FRAG_LOADING
buffer-controller.ts:692 [log] > [buffer-controller]: Updating Media Source duration to 600.015
base-stream-controller.ts:1761 [log] > [stream-controller]: FRAG_LOADING->IDLE
base-stream-controller.ts:556 [log] > [stream-controller]: Buffered main sn: initSegment of level 17 (frag:[NaN-NaN] > buffer:[0.017-7.267])
base-stream-controller.ts:727 [log] > [stream-controller]: Loading fragment 1 cc: 0 of [1-76] level: 17, target: 7.267
base-stream-controller.ts:1761 [log] > [stream-controller]: IDLE->FRAG_LOADING
transmuxer-interface.ts:227 [log] > [transmuxer-interface, main]: Starting new transmux session for sn: 1 p: -1 level: 17 id: 1
        discontinuity: false
        trackSwitch: true
        contiguous: false
        accurateTimeOffset: true
        timeOffset: 0
        initSegmentChange: true
base-stream-controller.ts:386 [log] > [stream-controller]: Loaded fragment 1 of level 17
transmuxer-interface.ts:379 [warn] > Adjusting initPTS by -7.25
onWorkerMessage @ transmuxer-interface.ts:379
TransmuxerInterface.onwmsg @ transmuxer-interface.ts:88
base-stream-controller.ts:1761 [log] > [stream-controller]: FRAG_LOADING->PARSING
stream-controller.ts:1288 [log] > [stream-controller]: Init video buffer, container:video/mp4, codecs[level/parsed]=[avc1.64002a/avc1.64002a]
audio-stream-controller.ts:128 [log] > [audio-stream-controller]: InitPTS for cc: 0 found from main: 10
transmuxer-interface.ts:379 [warn] > Adjusting initPTS by 2.8499999999999996
onWorkerMessage @ transmuxer-interface.ts:379
TransmuxerInterface.onwmsg @ transmuxer-interface.ts:88
audio-stream-controller.ts:128 [log] > [audio-stream-controller]: InitPTS for cc: 0 found from main: 12.85
transmuxer-interface.ts:379 [warn] > Adjusting initPTS by 2.4000000000000004
onWorkerMessage @ transmuxer-interface.ts:379
TransmuxerInterface.onwmsg @ transmuxer-interface.ts:88
audio-stream-controller.ts:128 [log] > [audio-stream-controller]: InitPTS for cc: 0 found from main: 15.25
transmuxer-interface.ts:379 [log] > [transmuxer.ts]: Flushed fragment 1 of level 17
transmuxer-interface.ts:379 [warn] > Adjusting initPTS by 2
onWorkerMessage @ transmuxer-interface.ts:379
TransmuxerInterface.onwmsg @ transmuxer-interface.ts:88
audio-stream-controller.ts:128 [log] > [audio-stream-controller]: InitPTS for cc: 0 found from main: 17.25
base-stream-controller.ts:1761 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:556 [log] > [stream-controller]: Buffered main sn: 1 of level 17 (frag:[0.000-2.850] > buffer:[0.017-7.267])
base-stream-controller.ts:1761 [log] > [stream-controller]: PARSED->IDLE
base-stream-controller.ts:727 [log] > [stream-controller]: Loading fragment 2 cc: 0 of [1-76] level: 17, target: 7.267
base-stream-controller.ts:1761 [log] > [stream-controller]: IDLE->FRAG_LOADING
base-stream-controller.ts:1761 [log] > [stream-controller]: FRAG_LOADING->PARSING

Chrome media internals output

No response

robwalch commented 1 year ago

Hi @otya128,

Progressive mode is listed as experimental and has limited support. Would you be willing to contribute a fix for this issue?

The test stream provided returns a 404 so I am unable to reproduce the issue.

otya128 commented 1 year ago

For example, when processing a segment like the one below:

segment1.mp4: duration=4s timeOffset=0s
  moof+mdat [0]: startDTS=0s duration=2s
  moof+mdat [1]: startDTS=2s duration=2s

https://github.com/video-dev/hls.js/blob/7ea23771aeada28e6d7ff5dafcf8697d3d22fc01/src/remux/passthrough-remuxer.ts#L250-L263 https://github.com/video-dev/hls.js/blob/7ea23771aeada28e6d7ff5dafcf8697d3d22fc01/src/remux/passthrough-remuxer.ts#L174-L188 isInvalidInitPts returns true when processing moof+mdat[1], initPTS is adjusted and playback gets stuck. https://github.com/otya128/hls.js/commits/fix-fmp4-progressive fixes by not adjusting initPTS except moof+mdat[0].

In non-progressive mode, moof+mdat[0] and moof+mdat[1] are processed at once, so there is no problem.

robwalch commented 1 year ago

Right. init PTS should be updated before the first segment append, and not before each chunk appended in progressive mode.

video-dev / hls.js