Smooth Streaming support for Dolby Digital Plus (EAC3/EC3)

sky-ermesombarros commented 2 years ago

Hello 👋

Is there any possibility adding supporting EAC3 (EC-3) for Smooth Streaming?

We've being doing some tests and noticed that RXPlayer does some filters and ignore EC-3 streams when playing Smooth Streaming, but even doing some changes to the player it doesn't with it correctly, for example:

We've tested adding EC-3 to supported mimetypes:

const MIME_TYPES : Partial<Record<string, string>> = {
  "EC-3": "audio/mp4",
  AACL: "audio/mp4",
  AACH: "audio/mp4",
  AVC1: "video/mp4",
  H264: "video/mp4",
  TTML: "application/ttml+xml+mp4",
  DFXP: "application/ttml+xml+mp4",
};

And also modified getAudioCodec to return ec-3 when identify that fourCC is EC-3:

export function getAudioCodecs(
  codecPrivateData : string,
  fourCC?: string
) : string {
  let mpProfile : number;
  if (fourCC === "AACH") {
    mpProfile = 5; // High Efficiency AAC Profile
  } else {
    mpProfile = isNonEmptyString(codecPrivateData) ?
      (parseInt(codecPrivateData.substring(0, 2), 16) & 0xF8) >> 3 :
      2;
  }
  if (fourCC === "EC-3") {
    return "ec-3";
  }
  if (mpProfile === 0) {
    // Return default audio codec
    return "mp4a.40.2";
  }
  return `mp4a.40.${mpProfile}`;
}

But even doing those steps the player fails with an BUFFER_APPEND_ERROR

image (5)

We suppose that the codec string we are using are correct as the player sets isSupported as true but even if we run MediaSource.isTypeSupported('audio/mp4;codecs="ec-3"') it returns true.

We are running tests on devices that supports surround sound, as PS5, Windows Edge and LG.

Also, running some tests with same stream but with Dash instead of Smooth it runs with no problems on those devices.

Can we get any ideas around it? Do we need to do anything on our fragments at the player side or use any different codec strings other than audio/mp4;codecs="ec-3?

peaBerberian commented 2 years ago

Hi,

A BUFFER_APPEND_ERROR happens when there's an error after pushing a segment. After re-looking at the Smooth code [1], my guess is that the issue is due to the fact that the initialization segment created and pushed by the RxPlayer still has an "mp4a" ISOBMFF box, which may not be what we want with DD+ (as as far as I know, this box is associated with aac codecs such as mp4a.40.2).

In DASH, initialization segments are downloaded and not created from scratch by using Manifest information like we do in Smooth, so there's no such problem there.

To fix it, we should probably construct the box related to ec-3 instead when the codec corresponds to dolby digital+. This may not be as complicated as it sounds - and may even be very straightforward - depending on the difference between the two boxes. I did not look into it too long for now, but to know how to implement it, we could look at the specifications if available, look at what a functional DASH DD+ initialization segment contains compared to a mp4 one and/or look at what compatible Smooth players are doing with the same contents (either by looking at their code if available, or by monkey-patching buffering API such as SourceBuffer.prototype.appendBuffer).

Do you want to do it? I also can try to implement it (I think it would even be a nice feature), but I have a lot of higher priority tasks for now.

[1] We can see that the smooth "SegmentLoader" ( in src/transports/smooth/segment_loader.ts) actually calls a function called createAudioInitSegment when encountering a request for an audio initialization segment, itself creating a segment containing an stsd box with an mp4a box inside.

sky-ermesombarros commented 2 years ago

Hi,

Thanks for your quick response, that really helped us. We are working on it right now and will update here as far as we have some conclusive results.

Here are some useful documentation/tools that are also helping us:

canalplus / rx-player

Smooth Streaming support for Dolby Digital Plus (EAC3/EC3) #1166