search

skymaze / pymediasoup

mediasoup python client
MIT License
26 stars 14 forks source link

let pymediasoup to consume h264 stream #13

Closed qwert3356 closed 7 months ago

qwert3356 commented 7 months ago

I am trying to let pymediasoup example to consume H264 stream, stream is from another mediasoup web client with url:forceH264=true,

but pymediasoup's device capability cannot have h264 codec with profile-level 4d001f after load server's capability response, (4d001f is created by mediasoup server's consumer, i added 4d001f 42e01f 4d0032 in mediasoup server's config before, but still no luck)

Then the server will not pass the video consumer to pymediasoup program,

could you give me a hint to resolve this problem? many thanks.

mediasoup server version: v3.8.3 pymediasoup version: v0.2.6 mediasoup web mediasoup-client version: 3.7.6

getRouterRtpCapability response from mediasoup server:

{
    "response": True,
    "id": 8379773,
    "ok": True,
    "data": {
        "codecs": [
            {
                "kind": "audio",
                "mimeType": "audio/opus",
                "clockRate": 48000,
                "channels": 2,
                "rtcpFeedback": [{"type": "transport-cc", "parameter": ""}],
                "parameters": {},
                "preferredPayloadType": 100,
            },
            {
                "kind": "video",
                "mimeType": "video/VP8",
                "clockRate": 90000,
                "rtcpFeedback": [
                    {"type": "nack", "parameter": ""},
                    {"type": "nack", "parameter": "pli"},
                    {"type": "ccm", "parameter": "fir"},
                    {"type": "goog-remb", "parameter": ""},
                    {"type": "transport-cc", "parameter": ""},
                ],
                "parameters": {"x-google-start-bitrate": 200000},
                "preferredPayloadType": 101,
            },
            {
                "kind": "video",
                "mimeType": "video/rtx",
                "preferredPayloadType": 102,
                "clockRate": 90000,
                "parameters": {"apt": 101},
                "rtcpFeedback": [],
            },
            {
                "kind": "video",
                "mimeType": "video/VP9",
                "clockRate": 90000,
                "rtcpFeedback": [
                    {"type": "nack", "parameter": ""},
                    {"type": "nack", "parameter": "pli"},
                    {"type": "ccm", "parameter": "fir"},
                    {"type": "goog-remb", "parameter": ""},
                    {"type": "transport-cc", "parameter": ""},
                ],
                "parameters": {"x-google-start-bitrate": 200000},
                "preferredPayloadType": 103,
            },
            {
                "kind": "video",
                "mimeType": "video/rtx",
                "preferredPayloadType": 104,
                "clockRate": 90000,
                "parameters": {"apt": 103},
                "rtcpFeedback": [],
            },
            {
                "kind": "video",
                "mimeType": "video/H264",
                "clockRate": 90000,
                "parameters": {
                    "packetization-mode": 1,
                    "level-asymmetry-allowed": 1,
                    "profile-level-id": "4d001f",
                    "x-google-start-bitrate": 200000,
                },
                "rtcpFeedback": [
                    {"type": "nack", "parameter": ""},
                    {"type": "nack", "parameter": "pli"},
                    {"type": "ccm", "parameter": "fir"},
                    {"type": "goog-remb", "parameter": ""},
                    {"type": "transport-cc", "parameter": ""},
                ],
                "preferredPayloadType": 105,
            },
            {
                "kind": "video",
                "mimeType": "video/rtx",
                "preferredPayloadType": 106,
                "clockRate": 90000,
                "parameters": {"apt": 105},
                "rtcpFeedback": [],
            },
        ],
        "headerExtensions": [
            {
                "kind": "audio",
                "uri": "urn:ietf:params:rtp-hdrext:sdes:mid",
                "preferredId": 1,
                "preferredEncrypt": False,
                "direction": "sendrecv",
            },
            {
                "kind": "video",
                "uri": "urn:ietf:params:rtp-hdrext:sdes:mid",
                "preferredId": 1,
                "preferredEncrypt": False,
                "direction": "sendrecv",
            },
            {
                "kind": "video",
                "uri": "urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id",
                "preferredId": 2,
                "preferredEncrypt": False,
                "direction": "recvonly",
            },
            {
                "kind": "video",
                "uri": "urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id",
                "preferredId": 3,
                "preferredEncrypt": False,
                "direction": "recvonly",
            },
            {
                "kind": "audio",
                "uri": "http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time",
                "preferredId": 4,
                "preferredEncrypt": False,
                "direction": "sendrecv",
            },
            {
                "kind": "video",
                "uri": "http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time",
                "preferredId": 4,
                "preferredEncrypt": False,
                "direction": "sendrecv",
            },
            {
                "kind": "audio",
                "uri": "http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01",
                "preferredId": 5,
                "preferredEncrypt": False,
                "direction": "recvonly",
            },
            {
                "kind": "video",
                "uri": "http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01",
                "preferredId": 5,
                "preferredEncrypt": False,
                "direction": "sendrecv",
            },
            {
                "kind": "video",
                "uri": "http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07",
                "preferredId": 6,
                "preferredEncrypt": False,
                "direction": "sendrecv",
            },
            {
                "kind": "video",
                "uri": "urn:ietf:params:rtp-hdrext:framemarking",
                "preferredId": 7,
                "preferredEncrypt": False,
                "direction": "sendrecv",
            },
            {
                "kind": "audio",
                "uri": "urn:ietf:params:rtp-hdrext:ssrc-audio-level",
                "preferredId": 10,
                "preferredEncrypt": False,
                "direction": "sendrecv",
            },
            {
                "kind": "video",
                "uri": "urn:3gpp:video-orientation",
                "preferredId": 11,
                "preferredEncrypt": False,
                "direction": "sendrecv",
            },
            {
                "kind": "video",
                "uri": "urn:ietf:params:rtp-hdrext:toffset",
                "preferredId": 12,
                "preferredEncrypt": False,
                "direction": "sendrecv",
            },
        ],
    },
}

rtp capability after "await self._device.load(ans['data'])" (then it will be send in join reqest) it does not have h264 codec

{
    "codecs": [
        {
            "mimeType": "audio/opus",
            "clockRate": 48000,
            "channels": 2,
            "rtcpFeedback": [],
            "parameters": {},
            "kind": "audio",
            "preferredPayloadType": 100,
        },
        {
            "mimeType": "video/VP8",
            "clockRate": 90000,
            "rtcpFeedback": [
                {"type": "nack", "parameter": ""},
                {"type": "nack", "parameter": "pli"},
                {"type": "goog-remb", "parameter": ""},
            ],
            "parameters": {},
            "kind": "video",
            "preferredPayloadType": 101,
        },
        {
            "mimeType": "video/rtx",
            "clockRate": 90000,
            "rtcpFeedback": [],
            "parameters": {"apt": 101},
            "kind": "video",
            "preferredPayloadType": 102,
        },
    ],
    "headerExtensions": [
        {
            "kind": "audio",
            "uri": "urn:ietf:params:rtp-hdrext:sdes:mid",
            "preferredId": 1,
            "preferredEncrypt": False,
            "direction": "sendrecv",
        },
        {
            "kind": "video",
            "uri": "urn:ietf:params:rtp-hdrext:sdes:mid",
            "preferredId": 1,
            "preferredEncrypt": False,
            "direction": "sendrecv",
        },
        {
            "kind": "video",
            "uri": "http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time",
            "preferredId": 4,
            "preferredEncrypt": False,
            "direction": "sendrecv",
        },
        {
            "kind": "audio",
            "uri": "urn:ietf:params:rtp-hdrext:ssrc-audio-level",
            "preferredId": 10,
            "preferredEncrypt": False,
            "direction": "sendrecv",
        },
    ],
    "fecMechanisms": [],
}

consumer info that corrspond to producer from mediasoup web client's web cam:

{
    "peerId": "dg4z4puz",
    "producerId": "a421ccc7-92e8-45ae-a89a-82724f28fa90",
    "id": "50f8d63d-cccc-4481-9f21-87ec7a4cff18",
    "kind": "video",
    "rtpParameters": {
        "codecs": [
            {
                "mimeType": "video/H264",
                "payloadType": 105,
                "clockRate": 90000,
                "parameters": {
                    "level-asymmetry-allowed": 1,
                    "packetization-mode": 1,
                    "profile-level-id": "4d001f"
                },
                "rtcpFeedback": [
                    {
                        "type": "transport-cc",
                        "parameter": ""
                    },
                    {
                        "type": "ccm",
                        "parameter": "fir"
                    },
                    {
                        "type": "nack",
                        "parameter": ""
                    },
                    {
                        "type": "nack",
                        "parameter": "pli"
                    }
                ]
            },
            {
                "mimeType": "video/rtx",
                "payloadType": 106,
                "clockRate": 90000,
                "parameters": {
                    "apt": 105
                },
                "rtcpFeedback": []
            }
        ],
        "headerExtensions": [
            {
                "uri": "urn:ietf:params:rtp-hdrext:sdes:mid",
                "id": 1,
                "encrypt": false,
                "parameters": {}
            },
            {
                "uri": "http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time",
                "id": 4,
                "encrypt": false,
                "parameters": {}
            },
            {
                "uri": "http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01",
                "id": 5,
                "encrypt": false,
                "parameters": {}
            },
            {
                "uri": "urn:3gpp:video-orientation",
                "id": 11,
                "encrypt": false,
                "parameters": {}
            },
            {
                "uri": "urn:ietf:params:rtp-hdrext:toffset",
                "id": 12,
                "encrypt": false,
                "parameters": {}
            }
        ],
        "encodings": [
            {
                "ssrc": 767530289,
                "rtx": {
                    "ssrc": 767530290
                },
                "maxBitrate": 2000000
            }
        ],
        "rtcp": {
            "cname": "9Ah+kJncI8VBDkqF",
            "reducedSize": true,
            "mux": true
        },
        "mid": "5"
    },
    "type": "simple",
    "appData": {
        "peerId": "dg4z4puz"
    },
    "producerPaused": false
}
qwert3356 commented 7 months ago

i checked the NativeRtpCapabilities from aiortc peerconnection

it has two h264 profile level supported: Baseline 3_1 and constrained_baseline 3_1

"fmtp": [
                {"payload": 98, "config": "apt=97"},
                {
                    "payload": 99,
                    "config": "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f",
                },
                {"payload": 100, "config": "apt=99"},
                {
                    "payload": 101,
                    "config": "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f",
                },
                {"payload": 102, "config": "apt=101"},
            ]

i change mediasoup server h264 codec config to 42e01f only then pymediasoup program can successfully consume h264 video frame

mediaCodecs :
    [
        {
            kind      : 'audio',
            mimeType  : 'audio/opus',
            clockRate : 48000,
            channels  : 2
        },
        {
            kind       : 'video',
            mimeType   : 'video/VP8',
            clockRate  : 90000,
            parameters :
            {
                'x-google-start-bitrate'  : 200000
            }
        },
        {
            kind       : 'video',
            mimeType   : 'video/VP9',
            clockRate  : 90000,
            parameters :
            {
                'x-google-start-bitrate'  : 200000
            }
        },
        {
            kind       : 'video',
            mimeType   : 'video/h264',
            clockRate  : 90000,
            parameters :
            {
                'packetization-mode'      : 1,
                'profile-level-id'        : '42e01f',
                'level-asymmetry-allowed' : 1,
                'x-google-start-bitrate'  : 200000
            }
        },
    ]