Audio Streaming and Backchannel Support

[EmpireofKings]

Hello,

Does the project have built in support for receiving the cameras audio stream ? or is that functionality not yet implemented ?

Also, does the project have back channel audio support ?

[EmpireofKings]

If the project does not support audio, then i would like to assist with the implementation, but before i begin i'm wondering what modifications would be required other than modifying the gstreamer pipeline.

Do we need to modify the browser client code to receive and parse the audio as well, or is that automagically handled ?

Do we need to grab the audio settings from the camera before streaming begins ? I'm very familiar with ONVIF, so if necessary, i can put the logic into grabbing the correct audio source and settings from the camera prior to the pipelines initialization .

[RSATom]

No there is no audio support at the moment.

Do we need to modify the browser client code to receive and parse the audio as well, or is that automagically handled ?

I'm not sure, but it's possible it will work without any changes. But even if not, it should not be a big problem to fix it.

Do we need to grab the audio settings from the camera before streaming begins ? I'm very familiar with ONVIF, so if necessary, i can put the logic into grabbing the correct audio source and settings from the camera prior to the pipelines initialization .

no, RTSP provides enough info to convert audio to WebRTC, the only problem here is codecs compatibility. It's possible it will be required to do audio transcoding to format supported by browsers.

[RSATom]

as start point you can look to https://github.com/WebRTSP/RtStreaming/blob/master/GstRtStreaming/GstReStreamer2.cpp

[EmpireofKings]

as start point you can look to https://github.com/WebRTSP/RtStreaming/blob/master/GstRtStreaming/GstReStreamer2.cpp

What does the full pipeline from uridecodebin -> webrtcbin look like ?

"uridecodebin ! h264parse config-interval=-1 ! rtph264pay pt=96 " Is this all of it ??

A while back i made a gstreamer rtsp-server that i used to relay camera feeds, and to relay the video/audio my pipeline configuration looked like this. (My program would automatically change audio depay/pay to the correct one required for the stream after discovering the format via a onvif request)

"( rtspsrc location=\"%s\" backchannel=onvif name=rtsprc_%d" " rtsprc_%d. ! capsfilter caps=application/x-rtp,media=video ! rtph264depay ! rtph264pay pt=96 name=pay0" " rtsprc_%d. ! capsfilter caps=application/x-rtp,media=audio ! rtppcmadepay ! rtppcmapay pt=97 name=pay1 )"

before i posed the question I tried modifying your pipeline, to this:

"uridecodebin ! tee name=t \ t. ! h264parse config-interval=-1 ! rtph264pay pt=96 \ t. ! capsfilter caps=application/x-rtp,media=audio ! rtppcmadepay ! rtppcmapay pt=97 name=pay1"

but, unfortunately, that didn't work, which is why im wondering if there's more to your pipeline, or if we are missing something on the browser side or in the transport side ?

[RSATom]

Maybe it will be simpler to start from GstReStreamer.cpp then.

GstReStreamer2 is intended to work with multiple clients at the same time, so it's a little bit more complicated (and you can find second part of pipeline in GstWebRTCPeer2.cpp)...

[RSATom]

I can prepare for you some basic demo based on GstReStreamer (something like https://github.com/WebRTSP/ClockServer) to simplify your life at the beginning a little bit, if you like...

[EmpireofKings]

I can prepare for you some basic demo based on GstReStreamer (something like https://github.com/WebRTSP/ClockServer) to simplify your life at the beginning a little bit, if you like...

I would appreciate that very much, i'm currently testing and modifying GstReStreamer.cpp.

I'll let you know if i get it with that, but still a basic audio demo would be much appreciated

[RSATom]

I'll try to find time for this soon...

[EmpireofKings]

Okay, so i'm back with some conclusions regarding the file GstReStreamer.cpp.

We need to modify the way you setup the pipeline, but first lets review what you are currently doing.

In the function GstReStreamer::prepare() you create a uridecodebin. Inside that function you set the caps of uridecodebin to only receive video at this line 49: g_object_set(decodebin, "caps", supportedCaps, nullptr);

After that this line 58: g_signal_connect(decodebin, "pad-added", G_CALLBACK(srcPadAddedCallback), this);

calls this function 78: GstReStreamer::srcPadAdded(GstElement* /*decodebin*/,GstPad* pad){} which creates the remainder of the pipeline elements and links decodebin to h264parse, at this line.

93: GstElement* out = gst_parse_bin_from_description( forceH264ProfileLevelId ? "h264parse config-interval=-1 ! rtph264pay pt=96 ! " "capssetter name=capssetter ! " "webrtcbin name=srcrtcbin" : "h264parse config-interval=-1 ! rtph264pay pt=96 ! " "webrtcbin name=srcrtcbin", TRUE, NULL);

---

So that ReStreamer can emit both audio and video, we need make the following changes:

Move the initialization of 'webrtcbin' to the function 'GstReStreamer::prepare()', this will allow us to connect both our audio and video bins to it.

Next we either need to provide a name for uridecodebin or simply add in a tee element that goes directly after uridecodebin. uridecodebin ! tee name=avtee or simply uridecodebin name=avtee

Next we need to allow the CAPS for audio AND video thus we need to remove line 49 and its associated logic.

Next we need to add a secondary call back that will allow us to connect a second srcpad from uridecodebin to a sinkpad for audioconverting.

Finally we need to modify both the videocallback(93) and our audiocallback string so that they end with the name of the webrtcbin, and begin with the name of the tee or uridecodebin element.

The audio creation string should look something like:

"avtee. ! audioconvert ! audioresample ! queue ! opusenc ! rtpopuspay ! queue ! application/x-rtp,media=audio,encoding-name=OPUS,payload=97 ! srcrtcbin."

and the video creation string should look something like:

"avtee. ! h264parse config-interval=-1 ! rtph264pay pt=96 ! srcrtcbin."

After that, everything should work just fine, BUT im not done implementing all of this just yet, i'll submit a pull request for the changes im making to GstReStreamer.cpp when im done, but i think i'll need you to implement these changes in GstReStreamer2.cpp

I referred to this example, go to line 398 for the pipeline string.

[RSATom]

Move the initialization of 'webrtcbin' to the function 'GstReStreamer::prepare()'

no, it's wrong, since GstReStreamer2 has multiple webrtcbin instances (1 per client). But overall it sounds right (except maybe some other minor issues)

[RSATom]

~FYI, I found GstReStreamer is broken in master. Trying to fix...~ fixed

[RSATom]

@EmpireofKings Please try https://github.com/WebRTSP/MinimalReStreamer

[EmpireofKings]

FYI, I found GstReStreamer is broken in master. Trying to fix...

That makes alot of sense, i kept running into issues with master earlier, so i ended up using a very old version of this repository.

(I cloned it sometime in 2021, im not sure exactly when, but its old)

I wasn't checking this comment thread or i wouldve just used your minimal example !

@EmpireofKings Please try https://github.com/WebRTSP/MinimalReStreamer

I previously made a comment that i had made progress, unfortunately, my attempted solution does not want to work.

I'll keep trying and keep you posted.

[EmpireofKings]

@RSATom quick question, is there a reason you chose to use uridecodebin over rtspsrc ?

[RSATom]

The reason is simple, with uridecodebin it's not limited to rtsp sources. And I don't have to build pipeline from scratch for every new source type.

[EmpireofKings]

The reason is simple, with uridecodebin it's not limited to rtsp sources. And I don't have to build pipeline from scratch for every new source type.

Makes sense, i decided to build out the pipeline and link it manually, so far i've gotten audio backchannel to work, but connecting the audio output from rtspsrc to webrtcbin is still eluding me.

However, i'm extremely close.

[EmpireofKings]

@RSATom Do you have a RTSP endpoint with audio ? If so, i can share what i've done so far, and maybe you'd be able to quickly point out whats wrong :laughing:

[RSATom]

You can install https://snapcraft.io/rtsp-test-server it gives h264 or vp8 video with opus audio.

I plan replace opus audio with g711 soon on h264 endpoints.

[EmpireofKings]

You can install https://snapcraft.io/rtsp-test-server it gives h264 or vp8 video with opus audio.

I plan replace opus audio with g711 soon on h264 endpoints.

Oh, nice, ill take a looksy at that

[RSATom]

Don't think it worth push sources here. It would be better if you do fork and push your changes there. It will be simpler track changes that way...

[EmpireofKings]

Don't think it worth push sources here. It would be better if you do fork and push your changes there. It will be simpler track changes that way...

I'll integrate, into the minimal restreamer and make a pull request :+1:

Unless you prefer a fork

[RSATom]

I don't know if it's possible to do pull request for subrepo... So just fork https://github.com/WebRTSP/RtStreaming and do your changes there...

[EmpireofKings]

I don't know if it's possible to do pull request for subrepo... So just fork https://github.com/WebRTSP/RtStreaming and do your changes there...

Yeah, i just pushed the changes to my fork.

[EmpireofKings]

Got it @RSATom

v=0
s=-
t=0 0
a=msid-semantic: WMS
m=video 57257 UDP/TLS/RTP/SAVPF 96
a=ice-options:trickle
a=setup:active
a=mid:video0
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001e
m=audio 55038 UDP/TLS/RTP/SAVPF 8
a=ice-ufrag:GQbl
a=ice-options:trickle
a=setup:active
a=mid:audio1
a=recvonly
a=rtcp-mux
a=rtpmap:8 PCMA/8000

Though i need to dig through WebRTSP.mjs to figure out whats causing this error when audio is available.

WebRTSP.js:304 Uncaught (in promise) DOMException: The play() request was interrupted by a new load request. https://goo.gl/LdLk22
_onTrack @ WebRTSP.js:304
_session.peerConnection.ontrack @ WebRTSP.js:297
wrappedCallback @ adapter-latest.js:2724

[RSATom]

@EmpireofKings why it's using WebRTSP.js? The latest version is .mjs for a long time...

[EmpireofKings]

@EmpireofKings why it's using WebRTSP.js? The latest version is .mjs for a long time...

A while back i messed up my proxy configuration and couldnt get .mjs files lol, so i just switched the names to .js for a quick fix.

[RSATom]

:thinking: but there is nothing on line 304 in the latest WebRTSP.mjs: https://github.com/WebRTSP/BrowserClient/blob/master/WebRTSP.mjs#L304

[EmpireofKings]

yeah i added some console.logs, but its this function.

I have audio=muted on the video element, but i think the problem is that audio is trying to connect before the video.

    _onTrack(event)
    {
304:        this._video.srcObject = event.streams[0];
    }

[RSATom]

ok, got it. Unfortunately I will not have time to look at it today...

[EmpireofKings]

ok, got it. Unfortunately I will not have time to look at it today...

no worries, i'm going to keep investigating, ill probably be able to figure it out after staring at it for a bit.

[EmpireofKings]

ok, got it. Unfortunately I will not have time to look at it today...

no worries, i'm going to keep investigating, ill probably be able to figure it out after staring at it for a bit.

I modified _onTrack and everything seems to be working fine.

I'm not sure if there are negative implications to doing this, but i am able to receive sound and video now.

Protip : Ensure that you drop your cache after changing WebRTSP.mjs :rofl:

 _onTrack(event)
    {
        if (event.receiver.track.kind == 'video'){
            this._video.srcObject = event.streams[0];
        }
    }

[RSATom]

:man_facepalming: it's my fault, obviously...

[EmpireofKings]

man_facepalming it's my fault, obviously...

No biggie, it was an easy oversight.

So now that audio streaming from (camera --> restreamer --> browser) works, i'm going to try and see what modifications are needed to send audio back from the browser to the cameras speaker ( camera <-- restreamer <-- browser) .

I'm thinking that we need to change webrtcbin to sendrcv instead of sendonly.

Then we can send the audio back from a browser mic and connect it to the audio backchannel call back that i implemented in my fork.

[EmpireofKings]

Just to update on my research, I've been looking at some examples.

The gstreamer receiving pipeline and its callbacks are straightforward, and using Navigator.getUserMedia() or MediaDevices.getUserMedia() to capture audio from a microphone on the browser is straightforward and well documented.

However, i'm not sure where, how, or if i should implement a JSON messenger / parser to handle the incoming sdp messages in the signalling service.

Such as these functions in the below .

on_server_message

soup_websocket_message_cb

on_server_message,

I've been looking at your signalling server and it seems to only be setup for sending messages, so we need to add in the required stuff to receive ? and in gstWebRTCPeer.cpp we need to add in another transceiver to receive the audio.

[RSATom]

1. There only 2 transceivers will be required - first for video, second for audio, and it doesn't matter if it will be send or sendrecv tranceiver
2. WebRTSP signalling already made bidirectional.
3. I've explicitly decided to not use JSON inside WebRTSP to avoid additional dependencies, and have all parsers self contained. My initial goal was create something universal as original RTSP is. And I was trying follow RTSP protocol as much as possible, but unfortunately WebRTC is too specific, and I was not able to use RTSP as is, and had to change logic. One more problem, I had no time to create docs for my vision of WebRTSP protocol...

But for 2. and 3. you are free to create your own signalling based on JSON (or anything else you like), and use only RtStreaming library.

[RSATom]

Just for note. If browser is initiator of session start - it uses following WebRTSP commands sequence:

1. DESCRIBE - to start media session and get sdp from server
2. SETUP - to exchange ice candidates
3. PLAY - to start playback and send client sdp

SETUP request can be issued by both server and client (since both can have ice candidates)

[RSATom]

One more note, If you will decide extend current implementation - presence of tranceivers and their direction (send/resv/sendresv) should be configurable. I mean it will be bad if client will connect to source without audio, but will be announced about audio channel. Or even worst if source doesn't have back channel but server will have recv or sendrecv audio tranceiver...

[EmpireofKings]

Apologies for the late reply, i took a break for the holidays.

But for 2. and 3. you are free to create your own signalling based on JSON (or anything else you like), and use only RtStreaming library.

I've decided to extend the current implementation, and not change or add anything to the signalling service.

Or even worst if source doesn't have back channel but server will have recv or sendrecv audio tranceiver...

The select-stream callback for rtspsrc automatically handles setting up a backchannel if one exists, if it does, i added a boolean to change the transceivers direction to sendrecv.

if (gst_structure_has_field (str, "a-sendonly")) {
        stream_id = idx;
        caps = gst_caps_new_empty ();
        str = gst_structure_copy (str);
        gst_structure_set_name (str, "application/x-rtp");
        gst_structure_filter_and_map_in_place (str, remove_extra_fields, NULL);
        gst_caps_append_structure (caps, str);
        setup_backchannel_shoveler (rtspsrc, caps);
        sendrecv = TRUE;
}

@RSATom so i've modified WebRTSP.mjs to ask for permission to use the microphone, and then added the microphones audio track/mediastream to the peerconnection in WebRTSP.mjs, afterward i set the Localdescription and send an offer to the server.

I used this repository as a guide to do so, as i did not want to make use of datachannels, like in this example, as using datachannels would force everyone to utilize gstreamer 1.16+.

For backend modifications i used this example as a guide to modify GstRTStreaming.cpp, i've also added two way transceivers to WebRTCPeer.cpp.

Unfortunately, the backend does not register the 'pad-added' signal when i connect the microphone in the browser.

To be honest, this is where i am stumped, i can't figure out how to properly send the browsers audio mediastream to webrtcbin and then have it fire the 'pad-added' signal.

I understand gstreamer pretty well, but i feel that i am stuck on the mechanics of setting up the ice connection with webrtcbin and the javascript peerconnection, do you know how to setup the connection from browser -> restreamer ?

If you know how to do that, and can provide me a working example, or some pseudocode describing the steps that should be taken on both ends, then I can take care of everything else and implement it all.

[RSATom]

To be honest, this is where i am stumped, i can't figure out how to properly send the browsers audio mediastream to webrtcbin and then have it fire the 'pad-added' signal.

Since it's server starts webrtc peer connection negotiation it should send sendrecv for audio in sdp. Please check it.

[RSATom]

Also, it's required to use getUserMedia on browser side to get access to mic.

[EmpireofKings]

Also, it's required to use getUserMedia on browser side to get access to mic.

Right, i've add this function to grab the microphone and add the mediatracks to the peer connection.

    async initiate_backchannel(){
        this._out_bound_stream = await navigator.mediaDevices.getUserMedia({video: false, audio: true});
        for (const track of this._out_bound_stream.getTracks()) {
          this.peerConnection.addTrack(track, this._out_bound_stream);
        }
    }

[RSATom]

can you please share sdp for both parties?

[EmpireofKings]

To be honest, this is where i am stumped, i can't figure out how to properly send the browsers audio mediastream to webrtcbin and then have it fire the 'pad-added' signal.

Since it's server starts webrtc peer connection negotiation it should send sendrecv for audio in sdp. Please check it.

Okay, here is the localdescription from the browser, which is generated after calling initiate_backchannel(), and provoking this command.

this._session.peerConnection.setLocalDescription(this._session.peerConnection.createOffer())

### localdescription From Browser

v=0
o=- +++++++++++++++++++7 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE video0 0
a=extmap-allow-mixed
a=msid-semantic: WMS EAM2yfNJoJe9bIs22kYXNobU45X1MLw2FSeE
m=video 49423 UDP/TLS/RTP/SAVPF 96 127 97 98 99 35 36 37 38 100 101 102 123 122 125 107 108 109 39 40 41 42 43 44 45 46 121 120 124 47
c=IN IP4 00.000.000.000
a=rtcp:9 IN IP4 0.0.0.0
a=candidate:1359984435 1 udp 2122262783 ****::****:****:****:****  37652 typ host generation 0 network-id 2
a=candidate:800974396 1 udp 2122194687 000.000.0.00 48108 typ host generation 0 network-id 1
a=candidate:3652034536 1 udp 1685987071 00.000.000.000 48108 typ srflx raddr 000.000.0.00 rport 48108 generation 0 network-id 1
a=candidate:529288131 1 tcp 1518283007 ****::****:****:****:****  9 typ host tcptype active generation 0 network-id 2
a=candidate:1631460044 1 tcp 1518214911 000.000.0.00 9 typ host tcptype active generation 0 network-id 1
a=candidate:3171713594 1 udp 41819903 00.000.000.000 49423 typ relay raddr 00.000.000.000 rport 48108 generation 0 network-id 1
a=ice-ufrag:y32k
a=ice-pwd:****************************
a=ice-options:trickle
a=fingerprint:sha-256 **************************
a=setup:actpass
a=mid:video0
a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 urn:3gpp:video-orientation
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e015
a=rtpmap:127 VP8/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=127
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:35 VP9/90000
a=rtcp-fb:35 goog-remb
a=rtcp-fb:35 transport-cc
a=rtcp-fb:35 ccm fir
a=rtcp-fb:35 nack
a=rtcp-fb:35 nack pli
a=fmtp:35 profile-id=1
a=rtpmap:36 rtx/90000
a=fmtp:36 apt=35
a=rtpmap:37 VP9/90000
a=rtcp-fb:37 goog-remb
a=rtcp-fb:37 transport-cc
a=rtcp-fb:37 ccm fir
a=rtcp-fb:37 nack
a=rtcp-fb:37 nack pli
a=fmtp:37 profile-id=3
a=rtpmap:38 rtx/90000
a=fmtp:38 apt=37
a=rtpmap:100 H264/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:123 rtx/90000
a=fmtp:123 apt=102
a=rtpmap:122 rtx/90000
a=fmtp:122 apt=96
a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125
a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:39 H264/90000
a=rtcp-fb:39 goog-remb
a=rtcp-fb:39 transport-cc
a=rtcp-fb:39 ccm fir
a=rtcp-fb:39 nack
a=rtcp-fb:39 nack pli
a=fmtp:39 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=4d001f
a=rtpmap:40 rtx/90000
a=fmtp:40 apt=39
a=rtpmap:41 H264/90000
a=rtcp-fb:41 goog-remb
a=rtcp-fb:41 transport-cc
a=rtcp-fb:41 ccm fir
a=rtcp-fb:41 nack
a=rtcp-fb:41 nack pli
a=fmtp:41 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=f4001f
a=rtpmap:42 rtx/90000
a=fmtp:42 apt=41
a=rtpmap:43 H264/90000
a=rtcp-fb:43 goog-remb
a=rtcp-fb:43 transport-cc
a=rtcp-fb:43 ccm fir
a=rtcp-fb:43 nack
a=rtcp-fb:43 nack pli
a=fmtp:43 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=f4001f
a=rtpmap:44 rtx/90000
a=fmtp:44 apt=43
a=rtpmap:45 AV1/90000
a=rtcp-fb:45 goog-remb
a=rtcp-fb:45 transport-cc
a=rtcp-fb:45 ccm fir
a=rtcp-fb:45 nack
a=rtcp-fb:45 nack pli
a=rtpmap:46 rtx/90000
a=fmtp:46 apt=45
a=rtpmap:121 red/90000
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=121
a=rtpmap:124 ulpfec/90000
a=rtpmap:47 flexfec-03/90000
a=rtcp-fb:47 goog-remb
a=rtcp-fb:47 transport-cc
a=fmtp:47 repair-window=10000000
m=audio 9 UDP/TLS/RTP/SAVPF 111 63 103 104 9 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:y32k
a=ice-pwd:***********************
a=ice-options:trickle
a=fingerprint:sha-256 ********************************
a=setup:actpass
a=mid:0
a=extmap:14 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:*****************************************
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:********** cname:VhwT/1+5/N2iUpmG
a=ssrc:********** msid:**************** 79*****-****-4**3-***e-*****0d3

### RemoteDescription From Browser

v=0
o=- ++++++++++++++++7 0 IN IP4 127.0.0.1
s=-
t=0 0
a=msid-semantic: WMS user@host-*************
m=video 58264 UDP/TLS/RTP/SAVPF 96
c=IN IP4 00.000.000.000
a=rtcp:54305 IN IP4 00.000.000.000
a=candidate:1 1 udp 2013266431 ****::****:****:****:**** 37713 typ host generation 0
a=candidate:2 1 tcp 1015023103 ****::****:****:****:**** 9 typ host tcptype active generation 0
a=candidate:3 1 tcp 1010828799 ****::****:****:****:**** 54255 typ host tcptype passive generation 0
a=candidate:4 1 udp 2013266430 00.000.000.000 48343 typ host generation 0
a=candidate:5 1 tcp 1015022079 00.000.000.000 9 typ host tcptype active generation 0
a=candidate:6 1 tcp 1010827775 00.000.000.000 46083 typ host tcptype passive generation 0
a=candidate:7 1 udp 2013266429 00.000.000.000 36954 typ host generation 0
a=candidate:8 1 tcp 1015022335 00.000.000.000 9 typ host tcptype active generation 0
a=candidate:9 1 tcp 1010828031 00.000.000.000 56635 typ host tcptype passive generation 0
a=candidate:1 2 udp 2013266430 ****::****:****:****:**** 33729 typ host generation 0
a=candidate:2 2 tcp 1015023102 ****::****:****:****:**** 9 typ host tcptype active generation 0
a=candidate:3 2 tcp 1010828798 ****::****:****:****:**** 48271 typ host tcptype passive generation 0
a=candidate:4 2 udp 2013266429 00.000.000.000 50148 typ host generation 0
a=candidate:5 2 tcp 1015022078 00.000.000.000 9 typ host tcptype active generation 0
a=candidate:6 2 tcp 1010827774 00.000.000.000 33849 typ host tcptype passive generation 0
a=candidate:7 2 udp 2013266428 00.000.000.000 40725 typ host generation 0
a=candidate:8 2 tcp 1015022334 00.000.000.000 9 typ host tcptype active generation 0
a=candidate:9 2 tcp 1010828030 00.000.000.000 34633 typ host tcptype passive generation 0
a=candidate:10 1 udp 1677722111 00.000.000.000 48343 typ srflx raddr 00.000.000.000 rport 48343 generation 0
a=candidate:10 2 udp 1677722110 00.000.000.000 50148 typ srflx raddr 00.000.000.000 rport 50148 generation 0
a=candidate:11 1 tcp 847249919 00.000.000.000 9 typ srflx raddr 00.000.000.000 rport 9 tcptype active generation 0
a=candidate:11 2 tcp 847249918 00.000.000.000 9 typ srflx raddr 00.000.000.000 rport 9 tcptype active generation 0
a=candidate:12 2 tcp 843055614 00.000.000.000 33849 typ srflx raddr 00.000.000.000 rport 33849 tcptype passive generation 0
a=candidate:13 2 udp 503316990 00.000.000.000 54305 typ relay raddr 00.000.000.000 rport 54305 generation 0
a=candidate:12 1 tcp 843055615 00.000.000.000 46083 typ srflx raddr 00.000.000.000 rport 46083 tcptype passive generation 0
a=candidate:13 1 udp 503316991 00.000.000.000 58264 typ relay raddr 00.000.000.000 rport 58264 generation 0
a=candidate:14 1 udp 1677722110 00.000.000.000 36954 typ srflx raddr 00.000.000.000 rport 36954 generation 0
a=candidate:15 1 tcp 847250175 00.000.000.000 9 typ srflx raddr 00.000.000.000 rport 9 tcptype active generation 0
a=candidate:16 1 tcp 843055871 00.000.000.000 56635 typ srflx raddr 00.000.000.000 rport 56635 tcptype passive generation 0
a=candidate:17 1 udp 503316990 00.000.000.000 64716 typ relay raddr 00.000.000.000 rport 64716 generation 0
a=candidate:14 2 udp 1677722109 00.000.000.000 40725 typ srflx raddr 00.000.000.000 rport 40725 generation 0
a=candidate:15 2 tcp 847250174 00.000.000.000 9 typ srflx raddr 00.000.000.000 rport 9 tcptype active generation 0
a=candidate:16 2 tcp 843055870 00.000.000.000 34633 typ srflx raddr 00.000.000.000 rport 34633 tcptype passive generation 0
a=candidate:17 2 udp 503316989 00.000.000.000 63092 typ relay raddr 00.000.000.000 rport 63092 generation 0
a=candidate:13 1 udp 335544831 00.000.000.000 60835 typ relay raddr 00.000.000.000 rport 60835 generation 0
a=candidate:13 2 udp 335544830 00.000.000.000 51653 typ relay raddr 00.000.000.000 rport 51653 generation 0
a=candidate:17 1 udp 335544830 00.000.000.000 60582 typ relay raddr 00.000.000.000 rport 60582 generation 0
a=candidate:17 2 udp 335544829 00.000.000.000 50819 typ relay raddr 00.000.000.000 rport 50819 generation 0
a=ice-ufrag:**************************
a=ice-pwd:***************************
a=ice-options:trickle
a=fingerprint:sha-256 ****************************************
a=setup:actpass
a=mid:video0
a=sendrecv
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=fmtp:96 packetization-mode=1;profile-level-id=42c015;sprop-parameter-sets=Z00AHpY1QUB7TcFAQVAAAAMAEAAAAwHoQA==,aO4xsg==
a=ssrc:---------- cname:user@host-*********
a=ssrc:---------- msid:user@host-********** webrtctransceiver32
m=audio 57631 UDP/TLS/RTP/SAVPF 8
c=IN IP4 00.000.000.000
a=rtcp:53907 IN IP4 00.000.000.000
a=candidate:1 1 udp 2013266431 ****::****:****:****:**** 49701 typ host generation 0
a=candidate:2 1 tcp 1015023103 ****::****:****:****:**** 9 typ host tcptype active generation 0
a=candidate:3 1 tcp 1010828799 ****::****:****:****:**** 53463 typ host tcptype passive generation 0
a=candidate:4 1 udp 2013266430 00.000.000.000 59044 typ host generation 0
a=candidate:5 1 tcp 1015022079 00.000.000.000 9 typ host tcptype active generation 0
a=candidate:6 1 tcp 1010827775 00.000.000.000 40101 typ host tcptype passive generation 0
a=candidate:7 1 udp 2013266429 00.000.000.000 44881 typ host generation 0
a=candidate:8 1 tcp 1015022335 00.000.000.000 9 typ host tcptype active generation 0
a=candidate:9 1 tcp 1010828031 00.000.000.000 38503 typ host tcptype passive generation 0
a=candidate:1 2 udp 2013266430 ****::****:****:****:**** 46506 typ host generation 0
a=candidate:2 2 tcp 1015023102 ****::****:****:****:**** 9 typ host tcptype active generation 0
a=candidate:3 2 tcp 1010828798 ****::****:****:****:**** 33727 typ host tcptype passive generation 0
a=candidate:4 2 udp 2013266429 00.000.000.000 38349 typ host generation 0
a=candidate:5 2 tcp 1015022078 00.000.000.000 9 typ host tcptype active generation 0
a=candidate:6 2 tcp 1010827774 00.000.000.000 57243 typ host tcptype passive generation 0
a=candidate:7 2 udp 2013266428 00.000.000.000 40363 typ host generation 0
a=candidate:8 2 tcp 1015022334 00.000.000.000 9 typ host tcptype active generation 0
a=candidate:9 2 tcp 1010828030 00.000.000.000 40015 typ host tcptype passive generation 0
a=candidate:10 1 udp 1677722111 00.000.000.000 59044 typ srflx raddr 00.000.000.000 rport 59044 generation 0
a=candidate:11 1 tcp 847249919 00.000.000.000 9 typ srflx raddr 00.000.000.000 rport 9 tcptype active generation 0
a=candidate:12 1 tcp 843055615 00.000.000.000 40101 typ srflx raddr 00.000.000.000 rport 40101 tcptype passive generation 0
a=candidate:13 1 udp 503316991 00.000.000.000 57631 typ relay raddr 00.000.000.000 rport 57631 generation 0
a=candidate:10 2 udp 1677722110 00.000.000.000 38349 typ srflx raddr 00.000.000.000 rport 38349 generation 0
a=candidate:14 1 udp 1677722110 00.000.000.000 44881 typ srflx raddr 00.000.000.000 rport 44881 generation 0
a=candidate:11 2 tcp 847249918 00.000.000.000 9 typ srflx raddr 00.000.000.000 rport 9 tcptype active generation 0
a=candidate:15 1 tcp 847250175 00.000.000.000 9 typ srflx raddr 00.000.000.000 rport 9 tcptype active generation 0
a=candidate:12 2 tcp 843055614 00.000.000.000 57243 typ srflx raddr 00.000.000.000 rport 57243 tcptype passive generation 0
a=candidate:13 2 udp 503316990 00.000.000.000 53907 typ relay raddr 00.000.000.000 rport 53907 generation 0
a=candidate:16 1 tcp 843055871 00.000.000.000 38503 typ srflx raddr 00.000.000.000 rport 38503 tcptype passive generation 0
a=candidate:17 1 udp 503316990 00.000.000.000 49748 typ relay raddr 00.000.000.000 rport 49748 generation 0
a=candidate:14 2 udp 1677722109 00.000.000.000 40363 typ srflx raddr 00.000.000.000 rport 40363 generation 0
a=candidate:15 2 tcp 847250174 00.000.000.000 9 typ srflx raddr 00.000.000.000 rport 9 tcptype active generation 0
a=candidate:16 2 tcp 843055870 00.000.000.000 40015 typ srflx raddr 00.000.000.000 rport 40015 tcptype passive generation 0
a=candidate:17 2 udp 503316989 00.000.000.000 64450 typ relay raddr 00.000.000.000 rport 64450 generation 0
a=candidate:17 1 udp 335544831 00.000.000.000 52877 typ relay raddr 00.000.000.000 rport 52877 generation 0
a=candidate:13 1 udp 335544830 00.000.000.000 52610 typ relay raddr 00.000.000.000 rport 52610 generation 0
a=candidate:13 2 udp 335544830 00.000.000.000 60819 typ relay raddr 00.000.000.000 rport 60819 generation 0
a=candidate:17 2 udp 335544829 00.000.000.000 50308 typ relay raddr 00.000.000.000 rport 50308 generation 0
a=ice-ufrag:*********************
a=ice-pwd:***************
a=ice-options:trickle
a=fingerprint:sha-256 **************************************************************
a=setup:actpass
a=mid:audio1
a=sendrecv
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:8 PCMA/8000
a=rtcp-fb:8 nack pli
a=ssrc:---------- cname:user@host-*******
a=ssrc:---------- msid:user@host-******* webrtctransceiver33

Would you like me to fetch the sdp from gstreamer aswell ?

[RSATom]

a=mid:video0
a=sendrecv

It should be sendonly in server side sdp I believe... But don't think it affects audio channel...

So it's something missing in code on browser side... Unfortunately I don't have any minimal WebRTC example for browser. I need to think how to find the possible reason of issue...

[EmpireofKings]

a=mid:video0
a=sendrecv

It should be sendonly in server side sdp I believe... But don't think it affects audio channel...

So it's something missing in code on browser side... Unfortunately I don't have any minimal WebRTC example for browser. I need to think how to find the possible reason of issue...

I was thinking that i messed up creating the offer.

Also, i may need to add in a 'onnegotiationneeded' callback to WebRTSP.mjs, but i wasn't sure if it was necessary.

Or, i might have all the correct functions and callbacks, but the order-of-operations is incorrect and out of order ?

[RSATom]

Or, i might have all the correct functions and callbacks, but the order-of-operations is incorrect and out of order ?

yes it's possible, but it's hard to say where exactly the problem is... WebRTC is pretty hard to debug due to many steps in peer connection establishing...

[EmpireofKings]

Or, i might have all the correct functions and callbacks, but the order-of-operations is incorrect and out of order ?

yes it's possible, but it's hard to say where exactly the problem is... WebRTC is pretty hard to debug due to many steps in peer connection establishing...

Right, i'm still trying to understand it all. I'm currently trying to debug the connection by following this guide/infographic from MDN.

It looks like im doing everything right on the javascript side though ?

[RSATom]

In our case ReStreamer = Naomi

[EmpireofKings]

In our case ReStreamer = Naomi

ooh, ive been looking at it backwards then. I assumed priya would be restreamer, and naomi was the browser.

[RSATom]

yes, it's ReStreamer creates offer, but browser asks about it in DESCRIBE request. And ReStreamers sends offer as part of reply to DESCRIBE request. After that browser sends answer as part of PLAY request.