Hi,

We implemented a workaround for orientation issues when sending video from iOS, and thought we’d post it here in case it’s useful to others.

We support Firefox, and we record video tracks using PlainRtpTransport/ffmpeg. For both of these use cases, video received from iOS clients is not properly rotated. Firefox does not support the video-orientation rtp header extension. And ffmpeg doesn’t either (or, perhaps, we have failed to figure out how to pass the orientation signaling through to ffmpeg in a usable fashion.)

So, removing the urn:3gpp:video-orientation from the sdp generated by mediasoup is a good workaround for these orientation issues. (Removing the extension from the sdp forces the client to encode the video rotated, rather than signaling rotation out of band.)

Here’s how we disable the extension, prior to calling the mediasoup-client device.load() function:

  log('loading soup device', routerRtpCapabilities);
  // remove rtp header orientation extension for mobile Safari,
  // because neither Firefox nor ffmpeg support it. so we need
  // Safari to encode the video as rotated, rather than just
  // signal rotation changes
  if (getBrowserName() === 'Safari' && browserMobile_p()) {
    routerRtpCapabilities.headerExtensions =
      routerRtpCapabilities.headerExtensions.filter(
        (ext) => ext.uri !== 'urn:3gpp:video-orientation'
      );
  }
  await this.soupDevice.load({ routerRtpCapabilities });

In our tests, Android Chrome clients work fine, which suggests that Android Chrome is not actually using this header extension. (We plan to revisit this to test more heavily on Android, however, and will update this post if we learn anything new.)

– Kwin

Thanks. I've added a Tricks section to the website, however it will be published in the next days once we release a new version of mediasoup. It will be as follows (note that links won't work here in GitHub as they are relative to the mediasoup website):

mediasoup v3 Tricks

Here some tricks for mediasoup.

RTP Capabilities Filtering

Related issue:

Wrong received video orientation in Firefox and FFmpeg.

The mediasoup router.rtpCapabilities represent the RTP capabilities that the router supports. While the codecs list depends on the RouterOptions given by during the router creation, other RTP capabilities such as the headerExtensions are fixed and are basically a copy of the headerExtensions in the supportedRtpCapabilities.ts file of mediasoup.

For instance, some of those RTP header extensions can affect the behaviour of the client. A good example is the "urn:3gpp:video-orientation" extension which, if supported by both the client and mediasoup, will make the client to not rotate its sending video (for instance when moving the mobile from portrait to landscape) but, instead, set an orientation value into a RTP header extension. This is supported by Chrome and any libwebrtc based endpoint (such as libmediasoupclient).

The problem with this is that, if a receiving consumer (i.e. Firefox as per today or FFmpeg) does not support such a RTP header extension, when the sender rotates its video the receiver will not realize of it and will render the received video with an orientation of 90º or -90º. In order to avoid that problem here the trick:

Being in sender side (let's assume it supports the "urn:3gpp:video-orientation" extension), the application may call device.load() (mediasoup-client, i.e. Chrome) or device.load() (libmediasoupclient) with a filtered version of the router RTP capabilities by removing the problematic headers. For instance, when in Chrome running mediasoup-client it would be as follows:

// Let's get router RTP capabilities via our own app signaling.
let routerRtpCapabilities = await mySignaling.request("getRouterRtpCapabilities");

// Just for Chrome, Safari or any libwebrtc based browser.
if (supportsVideoOrientationHeaderExtension)
{
  // Remove the "urn:3gpp:video-orientation" extension so when rotating the
  // device, Chrome will encode rotated video instead of indicating the video
  // orientation in an RTP header extension.
  routerRtpCapabilities.headerExtensions = routerRtpCapabilities.headerExtensions.
    filter((ext) => ext.uri !== 'urn:3gpp:video-orientation');
}

// Finally apply the router RTP capabilities to the device.
await device.load({ routerRtpCapabilities });

versatica / mediasoup-website

Disabling rtp orientation header extension for mobile Safari #8

mediasoup v3 Tricks

RTP Capabilities Filtering