After update 2022.2.1 no more Nest cam live stream

kaelsaes commented 2 years ago

The problem

After update 2022.2.1 the Nest cam live stream doesn‘t work anymore.

What version of Home Assistant Core has the issue?

Newest

What was the last working version of Home Assistant Core?

Last 2021

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Google Nest

Link to integration documentation on our website

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

austinbeam commented 2 years ago

This is happening for me as well, noticing that the new Nest cams are still working but the original nest cams are not.

kenttse commented 2 years ago

I am also seeing this behaviour after updating to 2022.02. I only have the original nest cams (Indoor and Doorbell) so I can't test the new cams. I am seeing 404 errors as described in #60353. I downgraded back to 2021.12.10.

One thing to add, upon restarting HA, cameras are available for a period of time before becoming unavailable again.

teknomar7 commented 2 years ago

I echo @kenttse... I get feeds after restarting, then they (usually) just die out and this started with the 2022.02 update and I get 404 errors in the logs. I have a Nest Doorbell (original / hardwired) and the Nest Cam Outdoor IQ. The doorbell seems to be much more flakey than the IQ cam.

KroZpOz commented 2 years ago

Confirming the exact same issue as others v2022.02 with Dropcam Pro. Downgrade to 2021.12.10 resolves the 404 / Stream Unavailable errors.

probot-home-assistant[bot] commented 2 years ago

nest documentation nest source _{^{(message by IssueLinks)}}

probot-home-assistant[bot] commented 2 years ago

Hey there @allenporter, mind taking a look at this issue as it has been labeled with an integration (nest) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

allenporter commented 2 years ago

Hi, this issue was not filed with the fields from the component filled out, so this was not routed properly to me. Please fill out the template next time as this was sitting for 6 days because it was missing information in the bug template.

Can someone who is experiencing this include diagnostic infomation? logs?

allenporter commented 2 years ago

Also any details about the particular lovelace UI configuration may help. Plus debug data as described at the bottom of https://www.home-assistant.io/integrations/nest/#troubleshooting would be extremely helpful.

Acknowledged that multiple users see this, so its a real issue.

teknomar7 commented 2 years ago

The error I see in the logs after restarting HA is the following (not sure if the URL contains sensitive info, so it is truncated):

Logger: homeassistant.components.stream.stream.camera.entryway_camera
Source: components/stream/__init__.py:340
Integration: Stream ([documentation](https://www.home-assistant.io/integrations/stream), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+stream%22))
First occurred: 9:04:45 AM (1 occurrences)
Last logged: 9:04:45 AM

Error from stream worker: Error opening stream (HTTP_NOT_FOUND, Server returned 404 Not Found) rtsps://stream-ue1-bravo.dropcam.com:443/sdm_live_stream/CiUA...

Despite that, I see the camera images in my lovelace panel for a bit before I get another error. The only lovelace UI addons I have from HACS are Simple Thermostat and slider-row-entity. Generally speaking, after some time the card in lovelace will say something about a stream buffer reader error, then it will just be a black square with nothing in it. Naturally I can't reproduce the problem now when responding to you but I will update this comment with additional logs if/when the problem comes back up.

As a side note, I cannot "download diagnostics" from the entity itself... it starts a download in my browser, then fails. Not sure if that's a permission issue since I'm remotely trying to download the file from an Odroid N2+.

Edit: So, even when the cams disappear, the log message is the same that shows up in the HA logs... 404. The endpoint changes from ue1-bravo to uc2-bravo. My configuration in the config.yml looks like below which all just references secrets in another file of course:

# Nest Integration
nest:
  client_id: !secret bluehanest_client_id
  client_secret: !secret bluehanest_client_secret
  # "Project ID" in the Device Access Console
  project_id: !secret bluehanest_project_id
  # Provide the full path exactly as shown under "Subscription name" in Google Cloud Console
  subscriber_id: !secret bluehanest_subscriber_id

allenporter commented 2 years ago

Thanks, https://github.com/home-assistant/core/pull/66304 fixes the diagnostics errors, as I noticed that also. So it sounds like you are saying that 2022.2.1 can work for you, but it is flaky, which are the symptoms of https://github.com/home-assistant/core/issues/60353 -- basically sometimes it 404s, sometimes it doesn't etc. That is kind of a "known issue" i've reported to the SDM API folks, and we're waiting on a fix for.

What is confusing is folks saying it doesn't work on 2022.2.X and works on 2021.12.X when rolling back, which implies a home assistant code change introduced this problem, and i'm not aware of what might have happened there given this code has been unchanged for awhile.

One other thing that did change in 2022.2 is that we enabled ll_hls by default. Perhaps something there is breaking, but it would not be related to the 404 stream errors, it would be more about the frontend side of things most likely: See https://www.home-assistant.io/integrations/stream/#configuration-variables more. So basically, one thing we could rule out is if ll_hls: False helps with 2022.2.

teknomar7 commented 2 years ago

Yeah, when I say flakey though... I mean it drops the camera feed every couple mins and sometimes it comes back, other times it doesn't without a restart. This was definitely not the behavior prior to the 2022.2.x release. I would get these stream errors in the logs, but it would almost never be noticeable in the feed or if it was, it was less frequent / briefly.

I can try the ll_hls setting... is this just an independent block in the config.yml separate from the actual nest block?

kenttse commented 2 years ago

I've had ll_hls set to true in my configuration back in 2021.12.X. I just restarted with ll_hls: False and report back once it runs for a little while.

allenporter commented 2 years ago

Ok thanks @kenttse for ruling out LL HLS. Just to answer the question from @teknomar7 yes exactly, its:

stream:
    ll_hls: False

But sounds like this has been ruled out. Thanks.

@teknomar7 thank you, this extra detail is helpful. I'm trying to tell the difference between "does not ever work at all, totally broken" vs "works then stops working" or how long. Streams expire after 5 minutes, for example, and the url is refreshed. The previous issue with 404s seem to hit some people more than others, but didn't seem to be related to the home assistant build.

Other questions I am still interested in:

exactly which lovelace setting are people using? Some people use picture-glance cards with various settings (live vs auto) while other use the entity card, etc
Does changing Preload stream change the behavior? This opens the stream at start and offers different retry behavior

teknomar7 commented 2 years ago

Yeah... It's hard to say... I mean before the 2022.2 update, I'd say it worked fine like 90% of the time and might glitch out for a few seconds or the picture would freeze... but usually solve by just refreshing the browser. Now a few mins after the reboot, the streams usually drop out and either stay out until i restart HA or sometimes they come back intermittently.

Here's a snippet of my lovelace card:

      - type: vertical-stack
        cards:
          - type: picture-entity
            entity: camera.entryway_camera
            show_name: false
            show_state: false
            camera_view: live
          - type: vertical-stack
            cards:
              - type: picture-entity
                entity: camera.garage_camera
                camera_view: live
                show_name: false
                show_state: false

allenporter commented 2 years ago

I set up a few home assistant instances in different configurations and was able to get one camera in a weird state.

From Developer Tools >> States I was able to lookup the camera and see these attributes for a working camera:

access_token: XXX
model_name: Camera
brand: Google Nest
frontend_stream_type: hls
entity_picture: >-
  /api/camera_proxy/camera.front_yard?token=XXX
friendly_name: Front Yard
supported_features: 2

And then for a non-working camera I see this only:

entity_picture: >-
  /api/camera_proxy/camera.front_door?token=XXXX
friendly_name: Front Door
supported_features: 2

So i think the problem is the frontend_stream_type is missing, etc. Something with attribute updates. If someone else can confirm they see this also that would help.

allenporter commented 2 years ago

I think the problem may be related to https://github.com/home-assistant/core/pull/62294 which "improved" how stream unavailability is tracked in camera. I think what may happen is if the stream goes unavailable, then these state attributes are not refreshed here, then the frontend stream which relies on these attributes to decide which player to use no longer works.

My hunch is that enabling Preload Stream is a workaround but it may be that you can only set this is the stream is available (e.g. right after startup).

allenporter commented 2 years ago

I think the problem is actually https://github.com/home-assistant/core/pull/61647/files which changed stream availability behavior, but was supposed to just be a logging change.

allenporter commented 2 years ago

Sent https://github.com/home-assistant/core/pull/66334 which I think will address this, and marked for inclusion in the next patch release.

allenporter commented 2 years ago

Fix should be in 2022.2.6

KroZpOz commented 2 years ago

Upgraded to v2022.02.06 yesterday and issue is still present, i.e., stream breaks after a few hours of uptime with error "Error from stream worker: Error opening stream (HTTP_NOT_FOUND, Server returned 404 Not Found)" Experiencing issue with Dropcam Pro.

teknomar7 commented 2 years ago

I'd have to agree... Issue is still present for me too after the update.

There's actually two errors I see in the logs (URLs are truncated)...

Logger: homeassistant.components.stream.stream.camera.garage_camera
Source: components/stream/__init__.py:340
Integration: Stream ([documentation](https://www.home-assistant.io/integrations/stream), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+stream%22))
First occurred: 8:27:52 AM (12 occurrences)
Last logged: 10:10:44 AM

Error from stream worker: Error opening stream (HTTP_NOT_FOUND, Server returned 404 Not Found) rtsps://stream-uc2-bravo.dropcam.com:443/sdm_live_stream/CiUA....

and also...

Error from stream worker: Error opening stream (INVALIDDATA, Invalid data found when processing input) rtsps://stream-uc2-bravo.dropcam.com:443/sdm_live_stream/CiUA2

Downloading diagnostics works now though and is as follows:

{
  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2022.2.6",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.9.7",
    "docker": true,
    "arch": "aarch64",
    "timezone": "America/Detroit",
    "os_name": "Linux",
    "os_version": "5.10.98",
    "supervisor": "2022.01.1",
    "host_os": "Home Assistant OS 7.4",
    "docker_version": "20.10.9",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "meross_lan": {
      "version": "2.4.3",
      "requirements": []
    },
    "ge_home": {
      "version": "0.5.0",
      "requirements": [
        "gehomesdk==0.4.12",
        "magicattr==0.1.5"
      ]
    },
    "deebot": {
      "version": "1.1.2",
      "requirements": [
        "deebot-client==1.1.2",
        "numpy>=1.21.2"
      ]
    },
    "hacs": {
      "version": "1.22.0",
      "requirements": [
        "aiogithubapi>=21.11.0"
      ]
    }
  },
  "integration_manifest": {
    "domain": "nest",
    "name": "Nest",
    "config_flow": true,
    "dependencies": [
      "ffmpeg",
      "http",
      "media_source"
    ],
    "documentation": "https://www.home-assistant.io/integrations/nest",
    "requirements": [
      "python-nest==4.2.0",
      "google-nest-sdm==1.7.1"
    ],
    "codeowners": [
      "@allenporter"
    ],
    "quality_scale": "platinum",
    "dhcp": [
      {
        "macaddress": "18B430*"
      },
      {
        "macaddress": "641666*"
      },
      {
        "macaddress": "D8EB46*"
      },
      {
        "macaddress": "1C53F9*"
      }
    ],
    "iot_class": "cloud_push",
    "is_built_in": true
  },
  "data": {
    "subscriber": {
      "start": 1,
      "message_received": 239,
      "message_acked": 239
    },
    "devices": [
      {
        "data": {
          "name": "**REDACTED**",
          "type": "sdm.devices.types.CAMERA",
          "assignee": "**REDACTED**",
          "traits": {
            "sdm.devices.traits.Info": {
              "customName": "**REDACTED**"
            },
            "sdm.devices.traits.CameraLiveStream": {
              "maxVideoResolution": {
                "width": 640,
                "height": 480
              },
              "videoCodecs": [
                "H264"
              ],
              "audioCodecs": [
                "AAC"
              ],
              "supportedProtocols": [
                "RTSP"
              ]
            },
            "sdm.devices.traits.CameraImage": {
              "maxImageResolution": {
                "width": 1920,
                "height": 1200
              }
            },
            "sdm.devices.traits.CameraPerson": {},
            "sdm.devices.traits.CameraSound": {},
            "sdm.devices.traits.CameraMotion": {},
            "sdm.devices.traits.CameraEventImage": {}
          },
          "parentRelations": [
            {
              "parent": "**REDACTED**",
              "displayName": "**REDACTED**"
            }
          ]
        },
        "command": {
          "sdm.devices.commands.CameraLiveStream.GenerateRtspStream": 1,
          "sdm.devices.commands.CameraLiveStream.ExtendRtspStream": 49,
          "sdm.devices.commands.CameraEventImage.GenerateImage": 64,
          "fetch_image": 63
        },
        "event_media": {
          "event": 59,
          "event.new": 34,
          "event.fetch": 59,
          "fetch_image": 59,
          "fetch_image.save": 63,
          "event.notify": 50,
          "event.update": 25,
          "fetch_image.skip": 20,
          "event.fetch_error": 1
        }
      },
      {
        "data": {
          "name": "**REDACTED**",
          "type": "sdm.devices.types.DOORBELL",
          "assignee": "**REDACTED**",
          "traits": {
            "sdm.devices.traits.Info": {
              "customName": "**REDACTED**"
            },
            "sdm.devices.traits.CameraLiveStream": {
              "maxVideoResolution": {
                "width": 640,
                "height": 480
              },
              "videoCodecs": [
                "H264"
              ],
              "audioCodecs": [
                "AAC"
              ],
              "supportedProtocols": [
                "RTSP"
              ]
            },
            "sdm.devices.traits.CameraImage": {
              "maxImageResolution": {
                "width": 1920,
                "height": 1200
              }
            },
            "sdm.devices.traits.CameraPerson": {},
            "sdm.devices.traits.CameraSound": {},
            "sdm.devices.traits.CameraMotion": {},
            "sdm.devices.traits.CameraEventImage": {},
            "sdm.devices.traits.DoorbellChime": {}
          },
          "parentRelations": [
            {
              "parent": "**REDACTED**",
              "displayName": "**REDACTED**"
            }
          ]
        },
        "command": {
          "sdm.devices.commands.CameraLiveStream.GenerateRtspStream": 1,
          "sdm.devices.commands.CameraEventImage.GenerateImage": 187,
          "fetch_image": 187,
          "sdm.devices.commands.CameraLiveStream.ExtendRtspStream": 49
        },
        "event_media": {
          "event": 180,
          "event.new": 112,
          "event.fetch": 180,
          "fetch_image": 180,
          "fetch_image.save": 187,
          "event.notify": 180,
          "event.update": 68,
          "fetch_image.skip": 61
        }
      }
    ]
  }
}

allenporter commented 2 years ago

I want to call a distinction between sometimes still 404s (https://github.com/home-assistant/core/issues/60353) tracked there and is still an issue. Does that make sense? I am not claiming you'll never see a 404, that is still a problem on the nest side.

The issue here gets stuck unavailable and never gets unstuck.

teknomar7 commented 2 years ago

Mine works on restarting HA, then dies and doesn't seem to come back at all until another restart. That could be anywhere from a min to a few mins after restarting HA where the feed dies out. I haven't seen it work consistently beyond 10 mins after restarting HA.

Edit... one thing I'd call out. I noticed the documentation says not to use the subscriber_id where previously I don't recall it saying that (when I initially set it up). I can attempt to comment that out in my config.yml to see if it makes any difference.

allenporter commented 2 years ago

Are you using Preload stream? Can you confirm the state is no longer stuck in Unavailable?

kenttse commented 2 years ago

Ditto, updated to 2022.2.6 this morning. Cameras working after the restart and gets marked unavailable at some later point. During the unavailable time, my automations that call for a snapshot or record fire, but do not capture anything. However, the snapshot is captured in the media source.

teknomar7 commented 2 years ago

I am using preload stream. If i go to the devices tab and hover over the exclamation, it says unavailable.

kenttse commented 2 years ago

I have Preload stream on, the camera is still stuck in Unavailable. Not all cameras (1 doorbell and 2 indoor) are marked Unavailable:

Capture

teknomar7 commented 2 years ago

Interestingly, it won't work without subscriber id in the yaml even though the documentation implies it's preferred to not define it. Throws an error saying the parm is required. Perhaps I'd have to change something on the Google side for it to work that way. I set this up awhile back and haven't touched it since.

allenporter commented 2 years ago

If you already have it set up, don't change anything :) (You can't take subscriber out if you set it up with subscriber, that is for new users. We can discuss this on another issue in more detail

Let me see if I can explain what i think we're seeing:

Nest cams generally have an issue where they throw 404s for some reason e.g. https://github.com/home-assistant/core/issues/60353 and other developers see this also https://stackoverflow.com/questions/70086032/rtsp-streams-often-fail-with-404-on-describe-method -- its not specific to home assistant.
In 2022.2 I changed the behavior for all streams as part of https://github.com/home-assistant/core/issues/54659 so that when a Preload stream is retrying in the worker, it is now marked unavailable. The previous behavior is that the frontend would not tell you there was a problem it it would either show you a frozen image or it would show a spinning video icon
There was a bug where that would happen even if it was not retrying. When marked unavailable, you could never get out of that state ever again. (Fixed in https://github.com/home-assistant/core/issues/60353)

So i think if we turn up debug logging for stream:

logger:
  default: info
  logs:
    homeassistant.components.stream: debug

we should see that the worker is in fact retrying with this log line https://github.com/home-assistant/core/blob/17a732197bb4f5b5a9b139bc7bdde707ffd37a02/homeassistant/components/stream/__init__.py#L360 --

I think there may still be another bug here too, i was able to see this myself on one of my home assistant instances that I left on overnight.

Thanks, all, i think i have enough info to dig some more.

allenporter commented 2 years ago

So far after 25 minutes what I am observing is one of my cameras will see 404s as we know (unfortunately), the stream marks itself as Unavailable (correct) while the worker detected a failure , it will retry with messages like Restarting stream worker in 10 seconds then Restarting stream worker in 20 seconds, etc. and then recovers on its own.

One hack we may want to do is always regenerate a new url when a 404 is found, which it currently does not do.

kaelsaes commented 2 years ago

I always get such an error message: 20220212_182936

allenporter commented 2 years ago

Yes, it uaed to silently break now it tells you it's broken.

kaelsaes commented 2 years ago

20220212_183224

allenporter commented 2 years ago

We may add client side retries in the frontend in https://github.com/home-assistant/frontend/issues/11018

My impression is this is not a regression in 2022.2, but something we can still try to improve.

kaelsaes commented 2 years ago

Allen, which Nest subscription is needed for a live stream? Is the cheap one sufficient?

allenporter commented 2 years ago

That may be more appropriate for a community forum discussion.

allenporter commented 2 years ago

I'm removing this from the next milestone since I think there is not additional new regressions here. I think we can follow improving robustness against the 404s in general in https://github.com/home-assistant/core/issues/60353 and i've submitted a few other improvements that can help in 2022.3

tomkessler commented 2 years ago

I'm having these issues as well; I'm on 2022.2.6. Same behavior as others in terms of errors. And my snapshot automations have basically stopped working.

My lovelace view is intermittent. And very strangely, sometimes they show the feed even though the state is "Unavailable". [See redacted image and entity info from dev tools]

Here's the lovelace card:

type: picture-entity
entity: camera.front_door
show_state: true
show_name: true

allenporter commented 2 years ago

If you are seeing 404s in the error log for all cameras then it is expected they are unavailable as we "improved" error handling. Before the errors used to be silently ignored by the frontend and showing a stale picture. I've linked the big above tracking 404s on the nest sdm API side. My impression is this is not a home assistant bug.

Asdie: If you are taking snapshots in response to events, you may find the new media player apis as a better approach as It won't necessarily have the same 404 issues since it's not using the rtsp stream.

kenttse commented 2 years ago

Hi @allenporter, I still am seeing that the 2020.12.10 version behaviour is quite different from the 2021.02.x versions. On the old version, I would rarely get blank images (camera.snapshot) or recordings (camera.record), but they did happen intermittently (one failure every two weeks). With the newer versions, it is almost guaranteed that every camera will eventually go into this "unavailable" state (within a couple of hours) until I restart.

Would the "unavailable" state affect snapshot and record?

allenporter commented 2 years ago

Yes, we're explicitly marking the camera unavailable now when the stream itself is broken as requested in https://github.com/home-assistant/core/issues/54659 to have "correct" error handling, which seems to turn off some things for the entity (e.g. stream won't render)

The way this works today is we have two processes going:

worker stream: background thread opening the rtsp stream and reading everything
snapshot image: opens the rtsp image again and connects to the stream

I think part of the reason why the stream is getting a 404 may be because the stream is being opened twice, once by the snapshot and once by the worker. I have a theory that using a Picture Glance Card with live may be less likely to get a 404 since the stream is only used once, even though it may happen from time to time still. I'm actively updating the code to not make both of these requests and to always just use the last image from the worker stream https://github.com/home-assistant/core/pull/66427

kenttse commented 2 years ago

Not sure if this helps or not, but I turned on debug logging and let it run over several hours:

2022-02-14 15:51:13 INFO (MainThread) [homeassistant.components.stream.stream.camera.entryway_doorbell_camera] Started stream
2022-02-14 15:51:26 ERROR (stream_worker) [homeassistant.components.stream.stream.camera.entryway_doorbell_camera] Error from stream worker: Error opening stream (HTTP_NOT_FOUND, Server returned 404 Not Found)
2022-02-14 15:51:26 DEBUG (stream_worker) [homeassistant.components.stream.stream.camera.entryway_doorbell_camera] Restarting stream worker in 10 seconds
2022-02-14 15:55:43 DEBUG (MainThread) [homeassistant.components.stream.stream.camera.entryway_doorbell_camera] Updating stream source

And every 5 minutes I see it updating the stream source between 15:55 until 16:31: 2022-02-14 16:31:48 DEBUG (MainThread) [homeassistant.components.stream.stream.camera.entryway_doorbell_camera] Updating stream source

At this point, I see the 404 error and restarting of the stream source and it backing off, but no update to the stream source. Then at 18:18, it just starts the stream again:

2022-02-14 18:17:05 ERROR (stream_worker) [libav.rtsp] method DESCRIBE failed: 404 Not Found
2022-02-14 18:17:05 ERROR (stream_worker) [homeassistant.components.stream.stream.camera.entryway_doorbell_camera] Error from stream worker: Error opening stream (HTTP_NOT_FOUND, Server returned 404 Not Found) 
2022-02-14 18:17:05 DEBUG (stream_worker) [homeassistant.components.stream.stream.camera.entryway_doorbell_camera] Restarting stream worker in 350 seconds:
2022-02-14 18:18:53 INFO (MainThread) [homeassistant.components.stream.stream.camera.entryway_doorbell_camera] Started stream

allenporter commented 2 years ago

I did fix in https://github.com/home-assistant/core/pull/66426 the backoff timer when urls are refreshed so things recover faster. I am marking that for a patch release.

You can see the backoff time by turning up debug logging for the stream component to verify it's waiting due to retries/error.

teknomar7 commented 2 years ago

I'm not actually seeing the stream restart successfully... I see backoffs / restarting the stream, but it never recovers. Also, at some point I start seeing invalid data as well as the 404.

2022-02-15 08:54:37 INFO (Thread-13) [google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager] Observed non-terminating stream error 500 Received RST_STREAM with error code 2
2022-02-15 08:54:37 INFO (Thread-13) [google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager] Observed recoverable stream error 500 Received RST_STREAM with error code 2
2022-02-15 08:54:37 INFO (Thread-13) [google.api_core.bidi] Re-established stream
2022-02-15 08:54:37 INFO (Thread-ConsumeBidirectionalStream) [google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager] Observed non-terminating stream error 500 Received RST_STREAM with error code 2
2022-02-15 08:54:37 INFO (Thread-ConsumeBidirectionalStream) [google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager] Observed recoverable stream error 500 Received RST_STREAM with error code 2
2022-02-15 08:57:37 DEBUG (MainThread) [homeassistant.components.stream.stream.camera.entryway_camera] Updating stream source rtsps://stream-ue1-bravo.dropcam.com:443/sdm_live_stream/CiUA2<redacted>
2022-02-15 08:57:39 DEBUG (MainThread) [homeassistant.components.stream.stream.camera.garage_camera] Updating stream source rtsps://stream-uc2-bravo.dropcam.com:443/sdm_live_stream/CiUA2<redacted>

That goes on for awhile then you see invalid data...

2022-02-15 09:07:18 ERROR (stream_worker) [homeassistant.components.stream.stream.camera.garage_camera] Error from stream worker: Error opening stream (INVALIDDATA, Invalid data found when processing input) rtsps://stream-uc2-bravo.dropcam.com:443/sdm_live_stream/CiUA<redacted>
2022-02-15 09:07:18 DEBUG (stream_worker) [homeassistant.components.stream.stream.camera.garage_camera] Restarting stream worker in 20 seconds: rtsps://stream-uc2-bravo.dropcam.com:443/sdm_live_stream/CiUA<redacted>

Then after that, I get some different errors further down from libav.rtsp...

2022-02-15 09:11:50 ERROR (stream_worker) [libav.rtsp] method DESCRIBE failed: 404 Not Found
2022-02-15 09:11:50 ERROR (stream_worker) [homeassistant.components.stream.stream.camera.entryway_camera] Error from stream worker: Error opening stream (HTTP_NOT_FOUND, Server returned 404 Not Found) rtsps://stream-ue1-bravo.dropcam.com:443/sdm_live_stream/CiUA2<redacted>
2022-02-15 09:11:50 DEBUG (stream_worker) [homeassistant.components.stream.stream.camera.entryway_camera] Restarting stream worker in 40 seconds: rtsps://stream-ue1-bravo.dropcam.com:443/sdm_live_stream/CiUA<redacted>

allenporter commented 2 years ago

It should have recovered after the log about updating the stream at 8:57, until the error at 09:07:18. Are you saying it was unavailable during that time?

allenporter commented 2 years ago

Also what UI card? My theory is it's partially due to the image thumbnail using the normal entity card and less likely to happen with a live picture glance card.

teknomar7 commented 2 years ago

It never recovered despite what the log said there. If it did recover, it wasn't for long enough for it to display in lovelace as I never saw anything come back up after it disappeared (besides some errors in the frame where the stream would be, then it would just go to a grayed out card). I'm using the picture-entity card.

allenporter commented 2 years ago

I am going to rollback the "unavailable" behavior change for nest until the nest sdm api side is fixed. To be clear, you will still see 404 errors and flakiness until nest fixes the APIs, tracked in #60353, it just won't mark the stream unavailable.

kaelsaes commented 2 years ago

Hi @allenporter, I still am seeing that the 2020.12.10 version behaviour is quite different from the 2021.02.x versions. On the old version, I would rarely get blank images (camera.snapshot) or recordings (camera.record), but they did happen intermittently (one failure every two weeks). With the newer versions, it is almost guaranteed that every camera will eventually go into this "unavailable" state (within a couple of hours) until I restart.

Would the "unavailable" state affect snapshot and record?

Same behaviour here.

I‘m using the picture glance card which becomes a small blank card once a day. I need to restart HA to get back the live stream.

kenttse commented 2 years ago

I manually patched my installation with the changes in #66571 and restarted around 2 PM EST. So far the stream has been restarted and I have not seen the failed streams since then. I'll continue to leave it running and see if the "unavailable" fix fixes this.

Updated to 2022.2.7 and restart at 8 AM, looking good so far

home-assistant / core