Playback exception on cmaf vod stream which the first segment have emsg box

guoen21 commented 10 months ago

Version

Media3 1.2.0

More version details

No response

Devices that reproduce the issue

samsung SM-S911U1 (RFCW11VVC1X) Android 13

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Not tested

Reproduction steps

Play a cmaf vod stream which the first segment contains emsg box e.g. the stream have pre-roll ads

Expected result

Start and watch this video normally.

Actual result

For media v1.0.2 it works well and no any exception. But for media v1.2.0 sometimes we can not start the playback and throws exceiption, sometimes it start normally. playback exception

I guess this issue will occur when we have pending metadata samples within fmp4 extractor. Our video chunk contain video + emsg samples and audio chuck only contain audio samples. If the audio chuck is extracted first and then no pending metadata (when extract video chuck the timeadjuster had inited), but if the video chuck is extracted first and then we will have pending metadata samples on media3 v1.2.0 (no peding emsg on v1.0.2). I think for the former the playback will be normal but for the latter it will break the playback.

On v1.2.0 for the case video chunk is extracted first, FragmentedMp4Extractor#onEmsgLeafAtomRead() have following logic:

commit emsg sample via emsgTrackOutput.sampleData() and then pending emsg to pendingMetadataSampleInfos. The emsgTrackOutput just cache the sample in local buffer, not commit sample to data queue yet.
at the time of audio chunk is extracted it will commit the audio samples, and then output all pendingMetadataSampleInfos via emsgTrackOutput.sampleMetadata() with offset. e.g we have 4 pending emsg, and the offset of first emsg is 441. Here I think we should pass 0 instead of offset to delegate.sampleMetadata() call. It may cause the offset is set to -441 for the first sample of emsg queue and cause the exception while reading emsg samples.

Media

Our stream use ssai ad solution, cmaf drm stream, and diffcult to make it to be public for debug.

Bug Report

[ ] You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

CarlLindqvist commented 8 months ago

We have the exact same issue on live streams with server side midrolls. Here is an example stream which stitches 2 minute midrolls every 5 minutes. https://csm-e-tv4se2-eb.tls1.yospace.com/csm/live/581556538.m3u8?vimondId=20315823&yo.ap=https%3A%2F%2Flive.streaming.a2d.tv%2Fysproxy%2Fhttps%3A%2F%2Fvod.streaming.a2d.tv%2Fys%2F&yo.asd=true&yo.av=3&yo.br=false&yo.fr=true&yo.me=true&yo.pdt=true&yo.t.jt=750&yo.up=https%3A%2F%2Flive.streaming.a2d.tv%2Fcmaf%2Fhttp%3A%2F%2Fvarnish-live-director.a2d.tv%2Fgroup02%2Fyospace%2Fdvrtest%2Fys.isml%2F&project=freewheel&yo.lpa=dur&t=long_form%2CTV4Play%2CNyheter+%26+debatt%2CCastens+h%C3%B6rna%2Clive%2CLive+cmaf+test+TV4%2Ctekniktest%2Csub_svod%2Cdt_mms_target_1_v3_30_days%2CSEG9791%2Cdt_mms_target_3_v3_30_days%2Cdt_mms_target_1_v3_aggressive%2Cdt_mms_target_3_v3_base%2CSEG2662%2Cdt_mms_target_1_v3_base%2Cdt_mms_target_3_v3_aggressive%2Cns_st_mv-dev%2Ctv4play.se&s=TV4Play%2FNyheter+%26+debatt%2FCastens+h%C3%B6rna%2Flive&dcid=d918a1f3-0bcc-438b-a205-b2a87ad0daf6&pid=1153490&st=0%3A0%2C20%3A1&gdpr=1&gdpr_consent=CP1YjVgP1YjVgAcABBENAaEgAAAAAAAAAChQAAAAAAGBIBgACAAFgAVAA8ACCAGQAaABEAFKAP0AxQC8wHLhAAIAGQwAIAIgAcxAAIAIgAcxwAgABAAFwAMgBfAIQAREArICkx0A8ABYAFQAQQAyADQAIgAYgBSgDRAH6ARYAxQCZAF5gMsAcuQACgAIAAyAF8ArICkyEAMABYAYgDFEoAwACAAFgBEADEAYoBeZIAMABcADIAXwCsgKTAZYUgGgALAAqACCAGQAaABEADEAKUAaIA_QCLAGKAXmA5coAFAAuABkAL4BWQDtgAAA.YAAAAAAAAAAA

etanmtv commented 8 months ago

This issue seems to have been introduced with the Media3 1.1.0 release

marcbaechinger commented 8 months ago

Thanks everyone for reporting! We will look into this.

From the duplicate #1203 and the information here it looks like this was introduced with 1.1.0:

For media v1.0.2 it works well and no any exception.

From #1203:

Issue introduced in: Media3 1.1.0 Not reproducable with: Media3 1.0.0

Note: there is another test stream in #1203 for repro and to verify a fix when available.

CarlLindqvist commented 8 months ago

The test stream in #1203 is actually the same test stream.

The midrolls trigger the issue consistently, so as soon as you get an ad you should get the index out of bounds error, most probably triggered by the emsg info.

marcbaechinger commented 7 months ago

Thanks! We can repro this with the stream. The analysis from @guoen21 is to the point. When calling sampleMetadata immediately after sampleData, the offset needs to be set to 0 (zero). The code is there since a while and was working because it apparently never happened that multiple EMSG samples were pending. A change shipped with 1.1.0 made this more likely and together this caused the problems starting with 1.1.0.

A fix for this will be referenced here when it lands on GitHub.

androidx / media