Add support for go2rtc 2-way audio

[luzik]

As frigate-card supporting AlexxIT way of delivering content, please consider adding support for this new project featuring two way audio communication

https://github.com/AlexxIT/go2rtc

[dermotduffy]

Relevant FAQ entry that includes an example of how this could be done:

Q. What about lovelace card with support 2-way audio?

At this moment I am focused on improving stability and adding new features to go2rtc. Maybe someone could write such a card themselves. It's not difficult, I have [some sketches](https://github.com/AlexxIT/go2rtc/blob/master/www/webrtc.html).

As this is realistically quite a niche feature request, keeping this open for long-term exploration, or someone who wishes to contribute.

[luzik]

My personal use case is to use frigate-card with my Dahua VTO (video doorbell), so from a user perspective it would be awesome, to have 3 extra buttons on live stream available.

• sound on/off
• microphone on/off
• unlock door button

[dermotduffy]

Agree that would be fantastic for that usecase (which is less niche)!

PS: you can add the unlock button today via a custom element (in case you hadn't seen that).

[conorlap]

This would be a great feature!!

[felipecrs]

Just in case it matters for the sake of reference, the sip-hass-card manages to use microphone in the HA dashboard:

https://github.com/TECH7Fox/sip-hass-card

It works even when using the companion app. And I concur, two-way audio would be a killing feature for me to fully get rid of the Dahua app and use HA only.

[felipecrs]

• Refs https://github.com/blakeblackshear/frigate/issues/2515

[felipecrs]

Another use case:

All my cameras have 2-audio support ("pretendbly" the same kind of 2-way audio supported by go2rtc) and having the option to speak through them would allow me to surprise someone if spoten doing some doubtful.

[calisro]

• sound on/off
• microphone on/off

^ This! I have to currently use a card_mod popup to speak to the cameras.

[dermotduffy]

I'm excited about the doorbell usecase, so do plan to work on this. Adding @AlexxIT as an FYI and in case they have any alternatives or reasons this would be a bad idea. My plan is simply to start experimenting with the sketch they already prepared in the context of the Frigate card.

[calisro]

Hi. I wanted to link this issue with this enhancement to take into consideration when adding support for 2-way audio.

https://github.com/AlexxIT/go2rtc/issues/67

It shouldn't merely be a requirement to 'mute' the microphone but instead I believe it should be considered to disable/enable it instead so as not to tie up the buttons on doorbells and other devices while simply 'viewing' the cameras.

[AlexxIT]

It is important to know that you need to access the microphone BEFORE starting the stream on the client. As far as I know, you can't add a track to already started WebRTC connection. But you can stop microphone right after init:

https://github.com/AlexxIT/go2rtc/blob/12925a6bc5ed203582f465ef45ef4992270d8823/www/webrtc.html#L98-L105

Also, go2rtc cannot start a backchannel for an already running RTSP connection. It will turn it on the first time it connects to the camera.

[calisro]

I wonder how the amcrest app does it. The app will show a live video of the doorbell while keeping the actual doorbell button active which is good. The app user can select the microphone button and it looks like it enables it on the existing stream.

Perhaps it is doing this:

But you can stop microphone right after init

and then re-enabling it.

Perhaps with proper disable/enable of the microphones, the doorbell remains active. But in my testing simply disabling the microphone in the html didn't stop the doorbell button from becoming disabled.

[felipecrs]

Well, another option would be to reload the stream upon enabling/disabling microphone.

[AlexxIT]

Someone has to wireshark the original app.

[Daniel-dev22]

Someone has to wireshark the original app.

What would I be looking for?

[AlexxIT]

The moment when amcrest app enable microphone

[whinis]

I will be getting one of these tomorrow and I can wireshark it and see what is being sent

[whinis]

I have a dump of the connection. Whenever I attempt to activate the microphone there must be an encrypted call from the server to the camera that I cannot decode and same from app to server. However afterwords there is a POST from the camera to the server with /device//transfer-stream/talk/1/0/encrypt with the payload shown below. The addr is the AWS server ip its contacting

{ "addr" : "34.232.113.251", "assistStream" : "true", "channelId" : "0", "design" : "first", "deviceType" : "device", "encrypt" : "1", "latitude" : "\<hex>", "location" : "\<hex>", "longitude" : "\<hex>", "num" : 0, "port" : 9112, "talkType" : "talk", "timeLimit" : true, "tlsPort" : 9113, "type" : "RTSP", "url" : "\/device\/\<serial>\/transfer-stream\/talk\/encrypt", "videoLimit" : 0, "viewer" : "294577" } Shortly afterwords the server makes a OPTIONS request to the camera with the following string

OPTIONS rtsp://127.0.0.1:554/cam/realmonitor?channel=1&subtype=0&encrypt=1&talktype=talk&proto=Private3

I no nothing else obvious outside of these two calls.

EDIT: Outside of these calls everything else seems to be over a tunnel of some sort setup via http over UDP

[felipecrs]

I still think it's very reasonable for the moment if the card simply reload the stream when you enable/disable microphone.

And I can't wait for this feature, I expect that I will be able to get rid of my Dahua app for answering the doorbell with it and therefore use solely HA.

PS: I would recommend to move the VTO-specific discussion to the go2rtc repository.

[AlexxIT]

I'm wondering if the app makes a new RTSP connection to the camera when you press the microphone button or if it sends this request within the existing RTSP connection.

[whinis]

I can send you a dump if you want AlexxIT, I cannot decipher anything with my limited knowledge because it seems to be wrapped in a tunnel. and every packet except these and a few TLS packets are 1514 bytes in size

[akston]

This issue seems to have stagnated, is this still a planned feature? Is there some other solution people have turned to (that doesn't involve SIP) that I'm unaware of?

[dermotduffy]

This issue seems to have stagnated, is this still a planned feature? Is there some other solution people have turned to (that doesn't involve SIP) that I'm unaware of?

Absolutely. For now, I'm trying to get v4.1.0 (backlog) out the door. This feature is scheduled for v4.2. v4.1 includes basic "native" go2rtc support, which I am hoping is a step towards this feature.

[felipecrs]

Is there some other solution people have turned to (that doesn't involve SIP) that I'm unaware of?

Are you aware of the latest improvements on go2rtc and WebRTC card towards 2-way audio?

[whinis]

For now I have relegated to just opening a proxy to go2rtc run off frigate without any security and sending a link in notifications to my phone

[felipecrs]

I have not tried, but doesn't the webrtc card now support 2 way audio?

[felipecrs]

never mind, I think it only supports playing audio.

[cbuxton5200]

I would love to have this support as well to answer video doorbell calls. Just as an FYI, in order to get a browser to support a microphone, you need to access the go2rtc URL via HTTPS with a trusted certificate.

[dermotduffy]

OK -- initial experimental support has been added and for now I'm going to close this issue.

• The support will be in the next build, but if you want to try it out you can use these build artifacts.
• As predicted, I could not find a way to add the microphone stream to the existing webrtc connection, so when the microphone button is first held it resets the connection. It's not too bad.
• I found go2rtc / webrtc 2-way support to be quite troublesome (bug1, bug2), so before filing issues against the card please make sure you have successfully tested 2-way audio in go2rtc itself.

[felipecrs]

Wow, this is awesome! Can't wait to test.