Nest WebRTC streams are never extended

[amanbx]

The problem

I'm trying to use a picture entity card to view a camera stream from a wired Nest outdoor camera. What I expect to see is a video stream that runs for longer than 5 minutes.

What I observe is

• The video stream ends at around the 5 minute mark
• The debug logs show that GenerateWebRtcStream was issued originally
• The debug logs show that ExtendWebRtcStream was never called

Since it appears that the API to extend the streams is never called, the streams simply expire after 5 minutes.

What version of Home Assistant Core has the issue?

core-2023.1.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

nest

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

$ grep auth /config/home-assistant.log  | less -S

2023-01-25 00:08:46.303 DEBUG (MainThread) [google_nest_sdm.auth] request[post]=https://smartdevicemanagement.googleapis.com/v1/enterprises/---redacted---/devices/xxxxxx-redacted-xxxxx:
2023-01-25 00:08:46.304 DEBUG (MainThread) [google_nest_sdm.auth] request[post json]={'command': 'sdm.devices.commands.CameraLiveStream.GenerateWebRtcStream', 'params': {'offerSdp': 'v=0\r\no=- 2190656088944067648 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1 2\r\na=ex
2023-01-25 00:08:47.586 DEBUG (MainThread) [google_nest_sdm.auth] response={'results': {'answerSdp': 'v=0\r\no=- 0 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 2 1\r\na=msid-semantic: WMS 8754699608566044512/654407170 virtual-6666\r\na=ice-lite\r\nm=audio 19305 UDP/TLS/
2023-01-25 00:08:48.066 DEBUG (MainThread) [google_nest_sdm.auth] request[post]=https://smartdevicemanagement.googleapis.com/v1/enterprises/---redacted---/devices/xxxxxx-redacted-xxxxx:
2023-01-25 00:08:48.066 DEBUG (MainThread) [google_nest_sdm.auth] request[post json]={'command': 'sdm.devices.commands.CameraLiveStream.GenerateWebRtcStream', 'params': {'offerSdp': 'v=0\r\no=- 3790461637932505801 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1 2\r\na=ex
2023-01-25 00:08:49.411 DEBUG (MainThread) [google_nest_sdm.auth] response={'results': {'answerSdp': 'v=0\r\no=- 0 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 2 1\r\na=msid-semantic: WMS 8754699608566044512/654407170 virtual-6666\r\na=ice-lite\r\nm=audio 19305 UDP/TLS/
2023-01-25 00:08:55.834 INFO (MainThread) [homeassistant.components.analytics] Submitted analytics to Home Assistant servers. Information submitted includes {'uuid': '19937826ba2a41b496a8a9ea36274430', 'version': '2023.1.5', 'installation_type': 'Home Assistant OS', 'supervisor':
2023-01-25 00:18:22.923 DEBUG (MainThread) [google_nest_sdm.auth] request[post]=https://smartdevicemanagement.googleapis.com/v1/enterprises/---redacted---/devices/xxxxxx-redacted-xxxxx:
2023-01-25 00:18:22.923 DEBUG (MainThread) [google_nest_sdm.auth] request[post json]={'command': 'sdm.devices.commands.CameraLiveStream.GenerateWebRtcStream', 'params': {'offerSdp': 'v=0\r\no=- 3137601597490183523 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1 2\r\na=ex
2023-01-25 00:18:23.177 DEBUG (MainThread) [google_nest_sdm.auth] request[post]=https://smartdevicemanagement.googleapis.com/v1/enterprises/---redacted---/devices/xxxxxx-redacted-xxxxx:
2023-01-25 00:18:23.178 DEBUG (MainThread) [google_nest_sdm.auth] request[post json]={'command': 'sdm.devices.commands.CameraLiveStream.GenerateWebRtcStream', 'params': {'offerSdp': 'v=0\r\no=- 7374016660981834326 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1 2\r\na=ex
2023-01-25 00:18:24.283 DEBUG (MainThread) [google_nest_sdm.auth] response={'results': {'answerSdp': 'v=0\r\no=- 0 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 2 1\r\na=msid-semantic: WMS 9501391475213323991/654407170 virtual-6666\r\na=ice-lite\r\nm=audio 19305 UDP/TLS/
2023-01-25 00:18:25.008 DEBUG (MainThread) [google_nest_sdm.auth] response={'results': {'answerSdp': 'v=0\r\no=- 0 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 2 1\r\na=msid-semantic: WMS 9501391475213323991/654407170 virtual-6666\r\na=ice-lite\r\nm=audio 19305 UDP/TLS/

### Additional information

_No response_

[allenporter]

I'll need to think about how to best make this generic so that the frontend will ask for this to be reloaded in the background.

Also I wonder if it makes sense to restart the stream when it fails after 5 minutes anyway.

[home-assistant[bot]]

nest documentation nest source

[synergie91]

I would like to add that I am having the same issue.

[synergie91]

Bumping this one.

Use case for me is to have an always-on CCTV screen, presented on a tablet device. Due to the always-on nature, the stream stopping after 5 minutes requires a manual refresh which also can sometimes lead to hitting a limit on the Google API

[robert113289]

bumping this one also.

I've had the issue since i got my cams last year. supposedly there is a webrtc extend in sdm. i dont see any code for refreshing a webRTC stream in the handle_stream_refresh method

[allenporter]

Yes, acknowledge this is an issue. The trick is we need to call the extend API somehow knowing the stream is still actively being viewed. It means there need to be new APIs in the frontend talking to the backend -- or it just has to retry when it fails.

[EvanGrote]

I'm also encountering this issue trying to consume my Nest Doorbell camera feed in frigate. Strangely enough the video-only feed works fine but I have corruption issues when I include audio in the stream and transcode the included audio from OPUS to AAC

[ali-essam]

For battery powered cameras, ExtendWebRtcStream won't work, would there be a way to re-generate a new one automatically?

Or at least that the Picture Glance card refreshes after 5 minutes or on failure?

A WebRTC stream cannot be extended on battery-powered doorbells. If a stream on a doorbell needs to be viewed beyond the initial session length, stop the existing stream and generate a new one. https://developers.google.com/nest/device-access/api/doorbell-battery#extend_a_live_stream

[allenporter]

@ali-essam i was thinking the client could detect the error and initiate a new session (with some safety mechanism or rate limiting to not hammer the frontend)

[acrylicus]

Bumping this; I know many people suffering from the same thing

[splinteredlight]

I'm also experiencing this.

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[splinteredlight]

This issue is still unresolved on Core 2024.2.1

[omochi1224]

Has this not been implemented yet?

[kostecky]

This issue is unresolved. Is there any work planned on it, or is it unworkable?

[allenporter]

This is possible to solve. There was a separate proposal to make some WebRTC enhancements for ring doorbells in https://github.com/home-assistant/architecture/discussions/1040 and we made some progress exploring a solution that would also work for this issue.

Next steps would be to update that proposal with a concrete API changes needed to expose session ids to the frontend and allow extending sessions.

[gtalarico]

I was about to create an issue but was happy to see it's already been reported and is under review.

For context, I am streaming a few cameras into a tablet mounted on the wall. I wanted to share a hacky work around for anyone needing a short term solution. I am just blindly reloading the page after 5 min if it has the selected dashboard open.

# configuration.yaml
frontend:
  themes: !include_dir_merge_named themes
  extra_module_url:
    - /local/reload.js

# www/reload.js
const dashboardName = "dashboard-tablet"
let timerId = null;

const observeUrlChange = () => {

    if (document.location.href.includes(dashboardName)) {
        console.log(`already on dashboard timer set - will reload in 5 min}`);
        timerId = setTimeout(function() { 
            console.log("reloading page")
            location.reload(); 
        }, 300000);
    }

    console.log("setting up observer")
    let oldHref = document.location.href;
    const body = document.querySelector("body");
    const observer = new MutationObserver(mutations => {
    if (oldHref !== document.location.href) {
        oldHref = document.location.href;
        let newHref = document.location.href;
        console.log("page changed: " + newHref);
        if (newHref.includes(dashboardName)) {
            console.log(`timer set - will reload in 5 min}`);
            timerId = setTimeout(function() { 
                console.log("reloading page")
                location.reload(); 
            }, 300000);
        } else {
            if (timerId != null) {
                console.log(`timer cleared`);
                clearTimeout(timerId);
                timerId = null
            }
        }
    }
    });
    observer.observe(body, { childList: true, subtree: true });
};
observeUrlChange()

[ccdunder]

Orthogonally, it seems to me like it would be good to retry WebRTC streams that disconnect (in general, not just Nest). This would be a simple client side change (handle connectionstatechange events). WDYT @allenporter ?

I have a hacky version of this that would be straightforward to integrate properly into the frontend.

[allenporter]

Yes, I agree with that direction since it's an existing problem. Just need to make sure it has some kind of backoff / pause to avoid continuously hammering the server. I would be very happy to review

[jaffacake]

I'm also experiencing this problem, @ccdunder any chance you could share your solution, it's be awesome to get this working!

[ccdunder]

@allenporter Thanks!

@jaffacake I started using go2rtc for a different reason (reducing video latency as much as possible). It happens that go2rtc has fixed this problem in an even better way: properly extending the Nest stream (https://github.com/AlexxIT/go2rtc/issues/723) in addition to handling connectionstatechange.

As far as mitigating this issue in home-assistant native, here's my working proof-of-concept:

1. Open browser dev tools.
2. Select the <ha-web-rtc-player ...> element.
3. Execute this in console:

   
   console.assert($0.__hass.entities[$0.entityid].platform == 'nest')

peerConnection = $0._peerConnection; peerConnection.addEventListener(‘connectionstatechange’, () => { if (peerConnection.connectionState = ‘failed’) { $0._startWebRtc(); } }

To productionize this, my first thought is to add this event listener near the end of [`HaWebRtcPlayer._startWebRtc‎`](https://github.com/home-assistant/frontend/blob/618cd9d9e55a17db69bcb825e519138456a61722/src/components/ha-web-rtc-player.ts#L155). Then wrapping it in exponential backoff as @allenporter mentioned.

[jfgodman]

Any progress on this? Thanks!

[allenporter]

Yes, working on it.

Proposal drafted 3 weeks ago: https://github.com/home-assistant/architecture/discussions/1151 New session based webrtc APIs in progress: https://github.com/home-assistant/core/pull/127981

Nest support can come after that lands.

[allenporter]

Given we have progress on the webrtc sessions APIs which can enable this, I would like to keep this issue only on discussing extending streams for nest. General webrtc reliability and retries is out of scope of this discussion, and can happen elsewhere.