Video streaming gets stuck in streaming/broken state after ignoring for awhile, and cannot be stopped manually

ahaverty commented 1 year ago

Describe the bug

Doorbell video seems to never automatically end its streaming even after waiting for hours. The doorbell stop streaming buttons/services does work after if I manually end it a few minutes after starting a stream. But if I start a stream manually, and leave it running for awhile, it never automatically resets itself back to Idle, and seems to get stuck in a broken state, and I can't even end it manually with the service/lovelace button (I'm using the latest lovelace conditional view with Webrtc as per v6 readme)

To reproduce

Steps to reproduce the behavior:

Start stream by clicking the camera
Notice webrtc view works fine
Leave homeassistant/browser, without manually stopping the stream
Return hours later, to a broken/black webrtc screen
Notice clicking "stop" button throws an error and doesn't change state/lovelace view (see below)

Expected behavior

From what I understood, the eufy integration/add-on should listen to notifications from Eufy cloud, or have some kind of timeout to automatically set the device/sensors back to "Idle".

Doorbell device and sensors in home assistant would automatically understand the stream is no longer running, so that the conditional card in lovelace returns to the last image recorded.
Or, I can at least manually click the stop button so that it returns to the camera view with the last image.

Additional information

Eufy: v6.0.0 Home Assistant Installation Type (OS - Supvervised - Core): Supvervised Home Assistant Core Version: Supervisor 2022.12.1 Operating System 9.4 Frontend 20221213.1 - latest

Hardware Information;

Camera Model: T8200
Live Streaming Protocol (RTSP/P2P): P2P
Debug Logs

HA logs:

Logger: homeassistant.components.websocket_api.http.connection
Source: custom_components/eufy_security/eufy_security_api/api_client.py:345
Integration: Home Assistant WebSocket API ([documentation](https://www.home-assistant.io/integrations/websocket_api), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+websocket_api%22))
First occurred: 15:54:01 (2 occurrences)
Last logged: 15:59:54

[281472234584528] {'type': 'result', 'success': False, 'messageId': 'device.stop_livestream.ae3822ca85c04126b3d679442ebc8fef', 'errorCode': 'device_livestream_not_running'}
[281472234584528] {'type': 'result', 'success': False, 'messageId': 'device.stop_livestream.b13c16425119434789f1d55bb2e8a61b', 'errorCode': 'device_livestream_not_running'}
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 200, in handle_call_service
    await hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1745, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1782, in _execute_service
    await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 213, in handle_service
    await service.entity_service_call(
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 678, in entity_service_call
    future.result()  # pop exception if have
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 943, in async_request_call
    await coro
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 715, in _handle_entity_call
    await result
  File "/usr/src/homeassistant/homeassistant/components/button/__init__.py", line 116, in _async_press_action
    await self.async_press()
  File "/config/custom_components/eufy_security/button.py", line 52, in async_press
    await handler_func()
  File "/config/custom_components/eufy_security/eufy_security_api/camera.py", line 166, in stop_livestream
    await self.api.stop_livestream(self.product_type, self.serial_no)
  File "/config/custom_components/eufy_security/eufy_security_api/api_client.py", line 235, in stop_livestream
    await self._send_message_get_response(OutgoingMessage(command_type, command=command, serial_no=serial_no))
  File "/config/custom_components/eufy_security/eufy_security_api/api_client.py", line 345, in _send_message_get_response
    return await future
custom_components.eufy_security.eufy_security_api.exceptions.FailedCommandException: {'type': 'result', 'success': False, 'messageId': 'device.stop_livestream.ae3822ca85c04126b3d679442ebc8fef', 'errorCode': 'device_livestream_not_running'}

Add-on logs:

error stack:
• message_handler.ts:240 handle
    src/lib/device/message_handler.ts:240:31
• task_queues:96 processTicksAndRejections
    node:internal/process/task_queues:96:5
• server.ts:125 receiveMessage
    src/lib/server.ts:125:21
2023-01-04 02:41:21.888  WARN Station <redacted id> - Heartbeat check failed. Connection seems lost. Try to reconnect... 
2023-01-04 02:41:21.890  INFO  Disconnected from station <redacted id> 
2023-01-04 02:41:27.166  INFO  Connected to station <redacted id> on host 192.168.86.30 and port 23187 
2023-01-04 06:43:51.908  ERROR Message error 
 LivestreamNotRunningError  Livestream for device <redacted id> could not be stopped, because it is not running
error stack:
• message_handler.ts:240 handle
    src/lib/device/message_handler.ts:240:31
• task_queues:96 processTicksAndRejections
    node:internal/process/task_queues:96:5
• server.ts:125 receiveMessage
    src/lib/server.ts:125:21
2023-01-04 09:45:54.915  INFO  Client disconnected with ip: <redacted ip> port: 59040 code: 1006 reason: Abnormal Closure 
2023-01-04 10:01:23.957  INFO  Client disconnected with ip: <redacted ip> port: 48708 code: 1006 reason: Abnormal Closure 
2023-01-04 13:54:08.813  INFO  Client disconnected with ip: <redacted ip> port: 47860 code: 1000 reason: Normal Closure 
2023-01-04 14:24:37.568  WARN Station <redacted id> - AES key could not be decrypted! The entire stream is discarded. - Error: 
 Error  Error during decryption (probably incorrect key). Original error: Error: error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error
error stack:
• NodeRSA.js:301 module.exports.NodeRSA.$$decryptKey
    node_modules/node-rsa/src/NodeRSA.js:301:19
• NodeRSA.js:249 module.exports.NodeRSA.decrypt
    node_modules/node-rsa/src/NodeRSA.js:249:21
• session.ts:1304 handleDataBinaryAndVideo
    node_modules/eufy-security-client/src/p2p/session.ts:1304:63
• session.ts:1239 handleData
    node_modules/eufy-security-client/src/p2p/session.ts:1239:18
• session.ts:1132 parseDataMessage
    node_modules/eufy-security-client/src/p2p/session.ts:1132:26
• session.ts:965 handleMsg
    node_modules/eufy-security-client/src/p2p/session.ts:965:26
• session.ts:1814 <anonymous>
    node_modules/eufy-security-client/src/p2p/session.ts:1814:56
• node:events:513 emit
    node:events:513:28
• node:dgram:930 onMessage
    node:dgram:930:8
2023-01-04 14:24:43.401  ERROR Message error 
 LivestreamNotRunningError  Livestream for device <redacted id> could not be stopped, because it is not running
error stack:
• message_handler.ts:240 handle
    src/lib/device/message_handler.ts:240:31
• task_queues:96 processTicksAndRejections
    node:internal/process/task_queues:96:5
• server.ts:125 receiveMessage
    src/lib/server.ts:125:21
2023-01-04 15:54:01.517  ERROR Message error 
 LivestreamNotRunningError  Livestream for device <redacted id> could not be stopped, because it is not running
error stack:
• message_handler.ts:240 handle
    src/lib/device/message_handler.ts:240:31
• task_queues:96 processTicksAndRejections
    node:internal/process/task_queues:96:5
• server.ts:125 receiveMessage
    src/lib/server.ts:125:21
2023-01-04 15:59:54.283  ERROR Message error 
 LivestreamNotRunningError  Livestream for device <redacted id> could not be stopped, because it is not running
error stack:
• message_handler.ts:240 handle
    src/lib/device/message_handler.ts:240:31
• task_queues:96 processTicksAndRejections
    node:internal/process/task_queues:96:5
• server.ts:125 receiveMessage
    src/lib/server.ts:125:21

ahaverty commented 1 year ago

And perhaps helpful, I'm able to fix this by reloading the Eufy integration in home assistant> integrations. It seems to reset the states/device back to Idle, and allow me to start/stop streaming manually again.

I had the same issue for version lower than v5 also, but had too many other issues going on to give a proper report.

Am I wrong in assuming the stream/device should eventually automatically set itself back to idle after some kind of cloud notification or timeout?

Thanks!

fuatakgun commented 1 year ago

thanks for raising this, I was hoping that I could find a fix for this, before it is found out :)

I dont have a good solution yet, on it

ahaverty commented 1 year ago

Thanks @fuatakgun 😆

In the meantime, do you reckon I'd be wasting my time creating an automation that maybe forces a camera.stop (or integration reload) say 10 minutes after the camera starts streaming?

fuatakgun commented 1 year ago

No, it is clever thing to do actually:-)

ahaverty commented 1 year ago

Great! Would you have any insights into how long Eufy automatically kills a stream? I tried testing this myself awhile ago, but was surprised to see it live beyond 15 minutes. Perhaps it varies based on motion events etc? 🤔

I don't mind it stopping in HA really, but would like to avoid HA killing streams that happen to coincide with someone actually clicking the doorbell (My fiance is fairly reliant on previewing the doorbell via the Eufy app for example).

Cheers!

ahaverty commented 1 year ago

I'll see how I get on with this in the meantime 👍

alias: Stop camera stream after 15 minutes
trigger:
  - platform: state
    entity_id:
      - camera.doorbell
    to: streaming
    for:
      hours: 0
      minutes: 15
      seconds: 0
action:
  - service: camera.turn_off
    entity_id: camera.doorbell

(Updated after https://github.com/fuatakgun/eufy_security/issues/647#issuecomment-1371197855 suggestion :+1:)

fuatakgun commented 1 year ago

instead of using a delay, check if camera.doorbell is in streaming state for 15 minutes. by the way, stopping stream in HA would not interfere with eufy app. HA is acting as if another user is using another eufy app

JuhaFIN1 commented 1 year ago

Is this doorbell T8210 under same problem? I cannot get image from it even after restart or reload new beta Eufy under HA. i have homebase 3 + camera series 3 whick work fine now finally :) But doorbell shows white only after start stream from it.

eufy1 eufy2

ahaverty commented 1 year ago

Update: even with the automation stopping the camera after 15 minutes, I still hit this issue regularly. It seems like the camera is stopped, but the eufy/camera's other sensors are stuck in incorrect states.

So, even though nothing's streaming, and Webrtc is throwing errors about the video, when I click the stop button (camera: turn_off service) I get an error about the camera is apparently already stopped. When this is the case, camera.turn_on fixes things (I trigger that manually), so I've added both a stop and start button to my lovelace views.

For context on my use case: I have an old Android phone is kiosk mode by the door, so that we can view who's at the door before unlocking. Therefore, having these manual button clicks is not ideal, as it it could take 5+ seconds for it to get the video streaming on the android kiosk after clicking.

EDIT: I'll test things out, by dropping the delay from 15mins to 5mins in the meantime.

fuatakgun commented 1 year ago

https://github.com/fuatakgun/eufy_security/discussions/694

redwood996 commented 1 year ago

Update: even with the automation stopping the camera after 15 minutes, I still hit this issue regularly. It seems like the camera is stopped, but the eufy/camera's other sensors are stuck in incorrect states.

So, even though nothing's streaming, and Webrtc is throwing errors about the video, when I click the stop button (camera: turn_off service) I get an error about the camera is apparently already stopped. When this is the case, camera.turn_on fixes things (I trigger that manually), so I've added both a stop and start button to my lovelace views.

For context on my use case: I have an old Android phone is kiosk mode by the door, so that we can view who's at the door before unlocking. Therefore, having these manual button clicks is not ideal, as it it could take 5+ seconds for it to get the video streaming on the android kiosk after clicking.

EDIT: I'll test things out, by dropping the delay from 15mins to 5mins in the meantime.

Has your 5mins delay fixed the problem? I'm facing similar problem.

ahaverty commented 1 year ago

@redwood996 it hasn't worked no. I've since added two buttons to each state of my lovelace views. So if it's a blank video, I try both stop and start buttons, one usually gives me a popup error to say it's either already running or couldn't be stopped, but the alternative button consistently fixes the stream.

I'm using start and stop P2P, using camera on/off wasn't doing it for me for some reason (Wired 2k doorbell).

redwood996 commented 1 year ago

@ahaverty Thank you for the update. I have been dealing with a similar issue. Whenever I initiate the P2P stream, I notice that the "video queue size" keeps fluctuating. Instead of checking the "stream status", I have been using this as an indicator to confirm that the video is functioning. I have configured two automations in HomeAssistant. The first automation triggers the "start P2P stream" command if the "video queue size" remains unchanged for five seconds. The second automation is set up to stop and restart streaming if the "video queue size" exceeds 50. While this setup is effective during the day, it seems to fail at night. I am still exploring better ways to maintain a consistent streaming experience.

akeslo commented 1 year ago

@redwood996

This is an interesting approach, do you mind sharing the automation?

redwood996 commented 1 year ago

@akeslo

I have identified several scenarios where the doorbell streaming may fail unexpectedly. These include:

The doorbell video size may continue to increase without stopping, in which case you will need to stop and start the p2p stream.
The doorbell video size may become stuck at a small number (less than 10) indefinitely. To resolve this, simply press the start p2p stream button to initiate the stream again.
The doorbell device may become unavailable, requiring you to restart the eufy security add-on, reload the eufy integration, and press start p2p stream.

Since these issues occur at random intervals, I have created 3 automations in Home Assistant to automatically restart the streaming process. So far it has been working as expected.

Fix when video size is stuck

alias: Keep doorbell streaming
description: Fix when queue size < 1 for 10 s or size not changing any more
trigger:
- platform: numeric_state
entity_id: sensor.doorbell_video_queue_size
for:
  hours: 0
  minutes: 0
  seconds: 10
below: 1
- platform: template
value_template: >-
  {{ (as_timestamp(states.sensor.time.last_changed) | int) >
  (as_timestamp(states.sensor.doorbell_video_queue_size.last_changed) | int
  + 10) }}
condition: []
action:
- service: button.press
data: {}
target:
  entity_id: button.doorbell_stop_p2p_stream
enabled: false
- delay:
  hours: 0
  minutes: 0
  seconds: 2
  milliseconds: 0
enabled: false
- service: button.press
data: {}
target:
  entity_id: button.doorbell_start_p2p_stream
mode: single

Restart when video size keeps increasing over 100

alias: Restart Doorbell Stream
description: Restart doorbell streaming if video size is over 100.
trigger:
- platform: numeric_state
entity_id: sensor.doorbell_video_queue_size
for:
  hours: 0
  minutes: 0
  seconds: 0
above: 100
id: Doorbell Stream Size Watch
condition: []
action:
- service: button.press
data: {}
target:
  entity_id: button.doorbell_stop_p2p_stream
- delay:
  hours: 0
  minutes: 0
  seconds: 2
  milliseconds: 0
- service: button.press
data: {}
target:
  entity_id: button.doorbell_start_p2p_stream
mode: restart

Restart Eufy add on if becomes unavailable

alias: Restart Eufy add on
description: Restart Eufy add on if becomes unavailable
trigger:
- platform: state
entity_id:
  - sensor.doorbell_stream_status
attribute: unavailable
enabled: false
- platform: time_pattern
hours: /1
condition: []
action:
- service: hassio.addon_restart
data:
  addon: 3a26b21d_eufy_security_addon
- delay:
  hours: 0
  minutes: 0
  seconds: 10
  milliseconds: 0
- service: homeassistant.reload_config_entry
data: {}
target:
  device_id: 88a1dbb405b175abbdb68ad0c4f0968b
- delay:
  hours: 0
  minutes: 0
  seconds: 6
  milliseconds: 0
- service: button.press
data: {}
target:
  entity_id: button.doorbell_start_p2p_stream
mode: single

akeslo commented 1 year ago

@redwood996 this is incredible, thank you for this!

fuatakgun commented 1 year ago

https://github.com/fuatakgun/eufy_security/issues/672

fuatakgun commented 1 year ago

https://github.com/fuatakgun/eufy_security/issues/668

fuatakgun commented 1 year ago

https://github.com/fuatakgun/eufy_security/issues/790

Subcode commented 12 months ago

Is there still no fix for this? Even with the automations i still come back to a camera that is off. Sometimes automations just fail to restart the stream, and i also use the rtsp stream to record the feed so losing it is bad.

Wouldnt it be possible to restart the stream from within the addon when it goes down?

Just checked, for instance one of the automations that presses the start button when video queue gets stuck stays in the "Still running" position and never completes, so the stream is sometimes not restarted. This is probably because the camera keeps saying it is Streaming even when it isnt. Waiting a random amount longer and pressing start again does help. But there is no way to figure out when start can be pressed again. This however would be fixed if i could press stop first, so if it was possible to allow continue on error this would atleast fix automation.

spunkt123 commented 5 months ago

I experienced the same problem. I had to stop and start my Eufy cam every 5 Minutes because the video stream broke down.

Now i tried to change the settings in the Eufy security app: Under NAS(RTSP) i changed "Auf NAS gespeicherter Videotyp" from "Ereignisse" to "Fortlaufende Aufzeichnung" - Sorry i do not know the correct english description but i think its something like: "On NAS saved video types" from "Events" to "Continous Streaming"

Since i changed this i had not to restart my cam for 2 hours. I can refresh my post if this changes.

fuatakgun / eufy_security

Video streaming gets stuck in streaming/broken state after ignoring for awhile, and cannot be stopped manually #647