Mediasoup client returns invalid payloadType for H264 on React Native.

[arcinston]

Bug Report

IMPORTANT: Mediasoup client returns invalid payloadType for H264 on React Native.

Your environment

• Operating system: MacOS Sonoma 14.2.1
• Browser version: React Native
• npm version:9.8.1
• mediasoup version: 3.13.16 (latest)
• mediasoup-client version: 3.7.2 (latest)

Issue description

we recently upgraded mediasoup from 3.12.13 and mediasoup client from 3.6.98 to the latest versions . on Web devices everything worked fine but for React Native specifically the payloadType for H264 video specifically throws this error on mediasoup

i think it is not using the preffered payloadType it receives from the mediasoup device

to show the discripancy

below are the rtpCapabilities.codecs

[
  {
    "mimeType": "audio/opus",
    "kind": "audio",
    "preferredPayloadType": 100,
    "clockRate": 48000,
    "channels": 2,
    "parameters": { "minptime": 10, "useinbandfec": 1 },
    "rtcpFeedback": [
      { "type": "transport-cc", "parameter": "" },
      { "type": "nack", "parameter": "" }
    ]
  },
  {
    "mimeType": "video/VP8",
    "kind": "video",
    "preferredPayloadType": 102,
    "clockRate": 90000,
    "parameters": {},
    "rtcpFeedback": [
      { "type": "goog-remb", "parameter": "" },
      { "type": "transport-cc", "parameter": "" },
      { "type": "ccm", "parameter": "fir" },
      { "type": "nack", "parameter": "" },
      { "type": "nack", "parameter": "pli" }
    ]
  },
  {
    "mimeType": "video/rtx",
    "kind": "video",
    "preferredPayloadType": 103,
    "clockRate": 90000,
    "parameters": { "apt": 102 },
    "rtcpFeedback": []
  },
  {
    "mimeType": "video/VP9",
    "kind": "video",
    "preferredPayloadType": 104,
    "clockRate": 90000,
    "parameters": {},
    "rtcpFeedback": [
      { "type": "goog-remb", "parameter": "" },
      { "type": "transport-cc", "parameter": "" },
      { "type": "ccm", "parameter": "fir" },
      { "type": "nack", "parameter": "" },
      { "type": "nack", "parameter": "pli" }
    ]
  },
  {
    "mimeType": "video/rtx",
    "kind": "video",
    "preferredPayloadType": 105,
    "clockRate": 90000,
    "parameters": { "apt": 104 },
    "rtcpFeedback": []
  },
  {
    "mimeType": "video/H264",
    "kind": "video",
    "preferredPayloadType": 108,
    "clockRate": 90000,
    "parameters": {
      "level-asymmetry-allowed": 1,
      "packetization-mode": 1,
      "profile-level-id": "42e034"
    },
    "rtcpFeedback": [
      { "type": "goog-remb", "parameter": "" },
      { "type": "transport-cc", "parameter": "" },
      { "type": "ccm", "parameter": "fir" },
      { "type": "nack", "parameter": "" },
      { "type": "nack", "parameter": "pli" }
    ]
  },
  {
    "mimeType": "video/rtx",
    "kind": "video",
    "preferredPayloadType": 109,
    "clockRate": 90000,
    "parameters": { "apt": 108 },
    "rtcpFeedback": []
  }
]

and this is the rtpParameter generated for the same

{
  "codecs": [
    {
      "mimeType": "video/H264",
      "payloadType": 98,
      "clockRate": 90000,
      "parameters": {
        "level-asymmetry-allowed": 1,
        "packetization-mode": 1,
        "profile-level-id": "42e034"
      },
      "rtcpFeedback": [
        { "type": "goog-remb", "parameter": "" },
        { "type": "transport-cc", "parameter": "" },
        { "type": "ccm", "parameter": "fir" },
        { "type": "nack", "parameter": "" },
        { "type": "nack", "parameter": "pli" }
      ]
    },
    {
      "mimeType": "video/rtx",
      "payloadType": 99,
      "clockRate": 90000,
      "parameters": { "apt": 98 },
      "rtcpFeedback": []
    }
  ],
  "headerExtensions": [
    {
      "uri": "urn:ietf:params:rtp-hdrext:sdes:mid",
      "id": 4,
      "encrypt": false,
      "parameters": {}
    },
    {
      "uri": "urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id",
      "id": 10,
      "encrypt": false,
      "parameters": {}
    },
    {
      "uri": "urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id",
      "id": 11,
      "encrypt": false,
      "parameters": {}
    },
    {
      "uri": "http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time",
      "id": 2,
      "encrypt": false,
      "parameters": {}
    },
    {
      "uri": "http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01",
      "id": 3,
      "encrypt": false,
      "parameters": {}
    },
    {
      "uri": "urn:3gpp:video-orientation",
      "id": 13,
      "encrypt": false,
      "parameters": {}
    },
    {
      "uri": "urn:ietf:params:rtp-hdrext:toffset",
      "id": 14,
      "encrypt": false,
      "parameters": {}
    }
  ],
  "encodings": [
    {
      "active": true,
      "scalabilityMode": "L1T3",
      "scaleResolutionDownBy": 4,
      "maxBitrate": 300000,
      "maxFramerate": 15,
      "rid": "r0",
      "dtx": false
    },
    {
      "active": true,
      "scalabilityMode": "L1T3",
      "scaleResolutionDownBy": 2,
      "maxBitrate": 600000,
      "maxFramerate": 30,
      "rid": "r1",
      "dtx": false
    },
    {
      "active": true,
      "scalabilityMode": "L1T3",
      "scaleResolutionDownBy": 1,
      "maxBitrate": 9000000,
      "maxFramerate": 30,
      "rid": "r2",
      "dtx": false
    }
  ],
  "rtcp": { "cname": "", "reducedSize": true },
  "mid": "0"
}

as you can see the payloadType is 98 which is not accepted by mediasoup

this seems to be just specific to react native as the same h264 codec with the exact same profile-level-id works fine on web

in web the preffered payloadType comes to be 108 and the payloadType is 106 for the same H264 codec

[arcinston]

I am trying to debug the issue from my end . Looking at the code changes in mediasoup-client , hoping i find the bug

but your input on this would also help

[arcinston]

also tried using vp9

works fine on web

but on react native , mediasoup server throws this error

i think the payloadType 103 is not meant to be for vp9 , i think there is some mistake here

maybe the starting offset is wrong like it should start from 100 but is starting from 97 so all payloadTypes are mismatching

[jmillan]

Did you check the RtpParameters provided to transport.produce()?

[ibc]

The payload type in PRODUCER RTP parameters doesn't need to match the preferred payload type in router.rtpCapabilities or device.rtpCaoabilities (those are just used for consuming) so that's not the problem. Can you please print:

• router.rtpCapabilities
• device.rtpCapabilities
• device._extendedRtpCapabilities (private API)
• client generated rtpParameters?
• the rtpParameters given to mediasoup side transport.produce() method

[arcinston]

@jmillan yes the second JSON i have provided are the rtpParameters recieived on the "produce" event on the sendTransport for that particular producer

@ibc yes , i agree it is just a preffered parameter and does not need to match the payloadType

i am sharing the requested details in the next comment

[ibc]

If you feel brave enough, you can add a test in node/src/test/test-Producer.ts with the capabilities and RTP parameters you get to reproduce the problem.

[arcinston]

@ibc @jmillan

there is no fault in either the mediasoup-client nor mediasoup, and for that i am extremely sorry.

the issue was we moved all mediaserver <-> signalling <-> client interactions from json to protobufs

here the schema for parameters in codecs was map<string,string> but some values could be numbers . although we had take care of parsing we missed a case where the codec.parameters from the server which were numbers were not parsed into string before sending to the client . everything seemed to work fine on web hence we didnt catch this early enough but didnt specifically on react native that is when we dug deeper compared all json files at figured it out.

thank you for taking the time to entertain my problem.

again grateful for the support you give !!

[ibc]

Thanks for explaining it.