Kurento Media Server not sending relay candidates, although configured, unless ANSWER is received or OFFER is processed

neilyoung commented 2 years ago

Prerequisites

These are MANDATORY, otherwise the issue will be automatically closed.

[x] I agree to fill this issue template.
[x] I have read the Troubleshooting Guide and Support Instructions.

Issue description

I'm letting Kurento generate the OFFER and forward it to the client. Unless the client responds with an ANSWER, no TURN relay candidates are delivered by KMS. I came across this while hunting down a possible problem with my ANSWERS. In this particular scenario TURN relay candidates are vital, otherwise no connection can be established

Context

How to reproduce?

On demand

Expected & current behavior

Relay candidates should be send, regardless of the SDP exchange

(Optional) Possible solution

Info about your environment

About Kurento Media Server

Kurento version:
Kurento Media Server version: 6.16.0 Found modules: 'core' version 6.16.0 'elements' version 6.16.0 'filters' version 6.16.0
Server OS: 18.04 bionic
Installation method:

About your Application Server

Programming Language:
Kurento Client version:

About end-user clients

Device(s):
OS(es):
Browser(s):

Run these commands

cat /etc/lsb-release

kurento-media-server --version

dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'

github-actions[bot] commented 2 years ago

Hello @neilyoung! :wave: we're sorry you found a bug... so first of all, thank you very much for reporting it.

To know about progress, check in Triage. All issues are considered Backlog Candidates until work priorities align and the issue is selected for development. It will then become part of our official Backlog.

neilyoung commented 2 years ago

Here is a sample app which demonstrates the problem:

const kurento = require('kurento-client')

const test = async () => {
    try {
        let kurentoClient = await kurento('ws://localhost:8888/kurento', { failAfter: 5 })
        let pipeline = await kurentoClient.create('MediaPipeline')
        await pipeline.setLatencyStats(true)
        let webRtcEndpoint = await pipeline.create('WebRtcEndpoint')

        await Promise.all(
            [webRtcEndpoint.setMinVideoSendBandwidth(500),
            webRtcEndpoint.setMaxVideoSendBandwidth(5000),
            webRtcEndpoint.setMinVideoRecvBandwidth(500),
            webRtcEndpoint.setMaxVideoRecvBandwidth(5000)
            ])

        webRtcEndpoint.on('ConnectionStateChanged', (event) => {
            console.log(`ConnectionStateChanged ${JSON.stringify(event)}`)
        })
        webRtcEndpoint.on('MediaStateChanged', (event) => {
            console.log(`MediaStateChanged ${JSON.stringify(event)}`)
        })
        webRtcEndpoint.on('IceComponentStateChange', (event) => {
            console.log(`IceComponentStateChange state ${event.state}, streamId ${event.streamId}, componentId ${event.componentId}`)
        })
        webRtcEndpoint.on('NewCandidatePairSelected', (event) => {
            console.log(`NewCandidatePairSelected, server ${event.candidatePair.localCandidate}`)
            console.log(`NewCandidatePairSelected, client ${event.candidatePair.remoteCandidate}`)
        })
        webRtcEndpoint.on('IceCandidateFound', (event) => {
            console.log(`IceCandidateFound ${event.candidate.candidate}, sdpMid: ${event.candidate.sdpMid}`)
        })
        webRtcEndpoint.on('IceGatheringDone', (_) => {
            console.log(`IceGatheringDone`)
        })

        await webRtcEndpoint.connect(webRtcEndpoint)
        console.log(await webRtcEndpoint.generateOffer())
        await webRtcEndpoint.gatherCandidates()

    }
    catch (e) {
        console.error(e)
    }
}

Run npm -i kurento-client before.

No STUN or TURN relay candidates

neilyoung commented 2 years ago

STUN/TURN candidates are offered, if KMS answers an OFFER. ~~GenerateOffer also produces STUN/TURN candidates, if called from the outside.~~ ERRATA: STUN/TURN candidates are NEVER offered if generateOffer() is called and no ANSWER is provided.

PLEASE EXPLAIN

The example below works. In this case KMS is receiving an OFFER.

const kurento = require('kurento-client')

const sdpOffer ="v=0\no=- 5830389526708069212 2 IN IP4 127.0.0.1\ns=-\nt=0 0\na=group:BUNDLE 0 1\na=extmap-allow-mixed\na=msid-semantic: WMS\nm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 121 127 120 125 107 108 109 35 36 124 119 123 118 114 115 116\nc=IN IP4 0.0.0.0\na=rtcp:9 IN IP4 0.0.0.0\na=ice-ufrag:49lr\na=ice-pwd:69SWVA8nR9TBi1lw5+77GSfB\na=ice-options:trickle\na=fingerprint:sha-256 4D:87:BF:C7:10:89:4C:8B:69:2E:51:DF:18:88:38:C4:9F:6C:9A:E9:1D:7B:3A:33:98:86:63:76:34:C9:C6:49\na=setup:actpass\na=mid:0\na=extmap:1 urn:ietf:params:rtp-hdrext:toffset\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\na=extmap:3 urn:3gpp:video-orientation\na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\na=sendrecv\na=msid:- 03d43219-9dcd-44ff-83cf-d70bb16ceda0\na=rtcp-mux\na=rtcp-rsize\na=rtpmap:96 VP8/90000\na=rtcp-fb:96 goog-remb\na=rtcp-fb:96 transport-cc\na=rtcp-fb:96 ccm fir\na=rtcp-fb:96 nack\na=rtcp-fb:96 nack pli\na=rtpmap:97 rtx/90000\na=fmtp:97 apt=96\na=rtpmap:98 VP9/90000\na=rtcp-fb:98 goog-remb\na=rtcp-fb:98 transport-cc\na=rtcp-fb:98 ccm fir\na=rtcp-fb:98 nack\na=rtcp-fb:98 nack pli\na=fmtp:98 profile-id=0\na=rtpmap:99 rtx/90000\na=fmtp:99 apt=98\na=rtpmap:100 VP9/90000\na=rtcp-fb:100 goog-remb\na=rtcp-fb:100 transport-cc\na=rtcp-fb:100 ccm fir\na=rtcp-fb:100 nack\na=rtcp-fb:100 nack pli\na=fmtp:100 profile-id=2\na=rtpmap:101 rtx/90000\na=fmtp:101 apt=100\na=rtpmap:102 H264/90000\na=rtcp-fb:102 goog-remb\na=rtcp-fb:102 transport-cc\na=rtcp-fb:102 ccm fir\na=rtcp-fb:102 nack\na=rtcp-fb:102 nack pli\na=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f\na=rtpmap:121 rtx/90000\na=fmtp:121 apt=102\na=rtpmap:127 H264/90000\na=rtcp-fb:127 goog-remb\na=rtcp-fb:127 transport-cc\na=rtcp-fb:127 ccm fir\na=rtcp-fb:127 nack\na=rtcp-fb:127 nack pli\na=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f\na=rtpmap:120 rtx/90000\na=fmtp:120 apt=127\na=rtpmap:125 H264/90000\na=rtcp-fb:125 goog-remb\na=rtcp-fb:125 transport-cc\na=rtcp-fb:125 ccm fir\na=rtcp-fb:125 nack\na=rtcp-fb:125 nack pli\na=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f\na=rtpmap:107 rtx/90000\na=fmtp:107 apt=125\na=rtpmap:108 H264/90000\na=rtcp-fb:108 goog-remb\na=rtcp-fb:108 transport-cc\na=rtcp-fb:108 ccm fir\na=rtcp-fb:108 nack\na=rtcp-fb:108 nack pli\na=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f\na=rtpmap:109 rtx/90000\na=fmtp:109 apt=108\na=rtpmap:35 AV1X/90000\na=rtcp-fb:35 goog-remb\na=rtcp-fb:35 transport-cc\na=rtcp-fb:35 ccm fir\na=rtcp-fb:35 nack\na=rtcp-fb:35 nack pli\na=rtpmap:36 rtx/90000\na=fmtp:36 apt=35\na=rtpmap:124 H264/90000\na=rtcp-fb:124 goog-remb\na=rtcp-fb:124 transport-cc\na=rtcp-fb:124 ccm fir\na=rtcp-fb:124 nack\na=rtcp-fb:124 nack pli\na=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0032\na=rtpmap:119 rtx/90000\na=fmtp:119 apt=124\na=rtpmap:123 H264/90000\na=rtcp-fb:123 goog-remb\na=rtcp-fb:123 transport-cc\na=rtcp-fb:123 ccm fir\na=rtcp-fb:123 nack\na=rtcp-fb:123 nack pli\na=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640032\na=rtpmap:118 rtx/90000\na=fmtp:118 apt=123\na=rtpmap:114 red/90000\na=rtpmap:115 rtx/90000\na=fmtp:115 apt=114\na=rtpmap:116 ulpfec/90000\na=ssrc-group:FID 3072426728 2555767047\na=ssrc:3072426728 cname:YcidyPojhj4f6zMt\na=ssrc:3072426728 msid:- 03d43219-9dcd-44ff-83cf-d70bb16ceda0\na=ssrc:3072426728 mslabel:-\na=ssrc:3072426728 label:03d43219-9dcd-44ff-83cf-d70bb16ceda0\na=ssrc:2555767047 cname:YcidyPojhj4f6zMt\na=ssrc:2555767047 msid:- 03d43219-9dcd-44ff-83cf-d70bb16ceda0\na=ssrc:2555767047 mslabel:-\na=ssrc:2555767047 label:03d43219-9dcd-44ff-83cf-d70bb16ceda0\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\nc=IN IP4 0.0.0.0\na=ice-ufrag:49lr\na=ice-pwd:69SWVA8nR9TBi1lw5+77GSfB\na=ice-options:trickle\na=fingerprint:sha-256 4D:87:BF:C7:10:89:4C:8B:69:2E:51:DF:18:88:38:C4:9F:6C:9A:E9:1D:7B:3A:33:98:86:63:76:34:C9:C6:49\na=setup:actpass\na=mid:1\na=sctp-port:5000\na=max-message-size:262144\n"

const test = async () => {
    try {
        let kurentoClient = await kurento('ws://localhost:8888/kurento', { failAfter: 5 })
        let pipeline = await kurentoClient.create('MediaPipeline')
        await pipeline.setLatencyStats(true)
        let webRtcEndpoint = await pipeline.create('WebRtcEndpoint')
        await Promise.all(
            [   webRtcEndpoint.setMinVideoSendBandwidth(500),
                webRtcEndpoint.setMaxVideoSendBandwidth(5000),
                webRtcEndpoint.setMinVideoRecvBandwidth(500),
                webRtcEndpoint.setMaxVideoRecvBandwidth(500)
            ])

        webRtcEndpoint.on('ConnectionStateChanged', (event) => {
            console.log(`ConnectionStateChanged ${JSON.stringify(event)}`)
        })
        webRtcEndpoint.on('MediaStateChanged', (event) => {
            console.log(`MediaStateChanged ${JSON.stringify(event)}`)
        })
        webRtcEndpoint.on('IceComponentStateChange', (event) => {
            console.log(`IceComponentStateChange state ${event.state}, streamId ${event.streamId}, componentId ${event.componentId}`)
        })
        webRtcEndpoint.on('NewCandidatePairSelected', (event) => {
            console.log(`NewCandidatePairSelected, server ${event.candidatePair.localCandidate}`)
            console.log(`NewCandidatePairSelected, client ${event.candidatePair.remoteCandidate}`)
        })
        webRtcEndpoint.on('IceCandidateFound', (event) => {
            console.log(`IceCandidateFound ${event.candidate.candidate}, sdpMid: ${event.candidate.sdpMid}`)
        })
        webRtcEndpoint.on('IceGatheringDone', (event) => {
            console.log(`IceGatheringDone`)
        })

        await webRtcEndpoint.connect(webRtcEndpoint)

//        await webRtcEndpoint.generateOffer()
        let answer = await webRtcEndpoint.processOffer(sdpOffer)
        console.log(answer)
        await webRtcEndpoint.gatherCandidates()

    }
    catch (e) {
        console.error(e)
    }

}

test()

neilyoung commented 2 years ago

I don't even recall if this is still an issue or not....

j1elo commented 2 years ago

Yeah it still happened... this pointed me to a part of the code that had always been kind of "not sure if this is right or wrong but it works for now". But it was actually wrong, only for some cases such as the one described. Once changing how it works, it seems that it now works fine for both cases.

neilyoung commented 2 years ago

Thanks for the insight. Have a nice weekend

Kurento / bugtracker