Unable to liveview using Ring Doorbell Wired. INVALIDDATA

DiEVeXx commented 2 years ago

Hi, First, thanks for your service and for your time.

Describe the problem

I am not able to get the real time view with a wired ring doorbell, neither from VLC nor from Home Assistant, but I am able to get the last captured snapshot.

Describe your environment

I'm using Raspberry OS (Raspbian GNU/Linux 11 (bullseye))

I am running Home Assistant Core latest stable, with this service ring-mqtt (latest) and mosquitto (latest) inside a docker network with a reverse proxy to access from outside home to my home assistant.

These are the steps followed in the deployment:

docker network:

docker network create -d overlay --ip-range=172.16.0.0/16 proxy

compose file:

version: '3'
services:
  homeassistant:
    container_name: homeassistant
    image: "ghcr.io/home-assistant/home-assistant:stable"
    volumes:
      - ./config:/config
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    privileged: true
    networks:
      - proxy

  ring-mqtt:
    container_name: ring-mqtt
    restart: unless-stopped
    image: tsightler/ring-mqtt
    ports:
      - 8554:8554                      # Enable RTSP port for external media player access
      - 55123:55123
    volumes:
      - ./ring-mqtt:/data           # Mapping of local folder to provide persistent storage
    logging:                           #limit logs to 10m and 3 files
      options:
        max-size: 10m
        max-file: "3"
    networks:
      - proxy

  reverseproxy:
    container_name: duckDNS
    restart: unless-stopped
    image: linuxserver/swag:arm32v7-1.28.0
    cap_add:
      - NET_ADMIN
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./duckdns_config:/config
    environment:
      - PUID=1000
      - PGID=1000
      - VALIDATION=duckdns
      - DUCKDNSTOKEN=**************************
      - EMAIL=*********@gmail.com
      - URL=************.duckdns.org
      - TZ=Europe/Madrid
    networks:
      - proxy

  eclipse-mosquitto:
    container_name: mosquitto
    restart: unless-stopped
    image: eclipse-mosquitto
    volumes:
      - ./mosquitto/config/mosquitto.conf:/mosquitto/config/mosquitto.conf
      - ./mosquitto/data:/mosquitto/data
      - ./mosquitto/log:/mosquitto/log
    networks:
      - proxy

networks:
  proxy: 
    external: true

ring mqtt config.json:

{
    "mqtt_url": "mqtt://mosquitto@eclipse-mosquitto:1883",
    "mqtt_options": "",
    "livestream_user": "****",
    "livestream_pass": "****",
    "disarm_code": "",
    "enable_cameras": true,
    "enable_modes": false,
    "enable_panic": false,
    "hass_topic": "homeassistant/status",
    "ring_topic": "ring",
    "location_ids": [
        ""
    ]
}

mosquitto.conf:

persistence true
persistence_location /mosquitto/data/

user mosquitto

# Listen on all interfaces 
listener 1883

#Allow connection without authentication
allow_anonymous true

log_dest file /mosquitto/log/mosquitto.log
log_dest stdout

After deployment get the Ring token from the :55123 and wait mosquitto to get information from ring-mqtt, Then i go to the Home assistant devices, search this doorbell from MQTT integration and click on Sensor Info to get the RTSP and Snapshot URLS.

But, here is where problems starts.

Describe any steps you've taken to attempt to resolve the problem

I have tried in the ring-mqtt configuration:

No password but user.
No user or password.
Set user and password

Options used in Camera integration config:

Use the local IP address of my machine.
Use 127.0.0.1 and localhost as url
Use the name of the ring-mqtt service as URL (as I use in the rest of the services).
Use the IP address of the container as shown in the Info sensor attributes as Stream Source.
Use TCP, UDP, Multicast, Set the user and password in the Authentication as well as to carry it in the RTSP URL or not to put it.

Debug Logs

Ring-Mqtt Logs (Starting service):

docker logs -f ha_ring-mqtt.1.pxj5ewo98e4igt3rl9rgeece7 
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/ring-mqtt.sh
-------------------------------------------------------
| Ring-MQTT with Video Streaming                      |
| Docker Edition                                      |
|                                                     |
| For support questions please visit:                 |
| https://github.com/tsightler/ring-mqtt/discussions  |
-------------------------------------------------------
cont-init: info: /etc/cont-init.d/ring-mqtt.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun ring-mqtt (no readiness notification)
s6-rc: info: service legacy-services successfully started
-------------------------------------------------------
ring-mqtt.js version: 5.0.3
Node version v16.16.0
NPM version 8.10.0
git version 2.36.2
-------------------------------------------------------
Running ring-mqtt...
2022-08-29T19:58:19.111Z ring-mqtt Detected runmode: docker
2022-08-29T19:58:19.118Z ring-mqtt Configuration file: /data/config.json
2022-08-29T19:58:19.123Z ring-mqtt MQTT URL: mqtt://mosquitto:********@eclipse-mosquitto:1883
2022-08-29T19:58:23.328Z ring-mqtt Reading latest data from state file: /data/ring-state.json
2022-08-29T19:58:23.488Z ring-mqtt Attempting connection to Ring API using saved refresh token...
2022-08-29T19:58:24.298Z ring-mqtt Successfully established connection to Ring API using saved token
2022-08-29T19:58:24.303Z ring-mqtt Received updated refresh token
2022-08-29T19:58:24.304Z ring-mqtt Saving updated refresh token to state file
2022-08-29T19:58:25.367Z ring-mqtt Successfully saved updated state file: /data/ring-state.json
2022-08-29T19:58:26.300Z ring-mqtt Attempting connection to MQTT broker...
2022-08-29T19:58:26.700Z ring-mqtt MQTT connection established, processing Ring locations...

After Processing Ring Locations the service get my location, my device and start to publish data to mosquitto.

Logs when i try to connect to live view in Home Assistant (using hostname IP):

2022-08-29 23:00:39.723 ERROR (stream_worker) [homeassistant.components.stream.stream.test_stream] Error from stream worker: Error opening stream (INVALIDDATA, Invalid data found when processing input) rtsp://****:****@192.168.100.4:8554/343ea4da34a8_live
2022-08-29 23:00:54.860 ERROR (stream_worker) [homeassistant.components.stream.stream.test_stream] Error from stream worker: Error opening stream (INVALIDDATA, Invalid data found when processing input) rtsp://****:****@192.168.100.4:8554/343ea4da34a8_live

Logs from Ring-Mqtt service after try to get

2022-08-29T21:00:34.599Z ring-rtsp INF [RTSP] [conn 10.0.0.2:35240] opened
2022-08-29T21:00:34.613Z ring-rtsp INF [path 343ea4da34a8_live] runOnDemand command started
2022-08-29T21:00:34.631Z ring-rtsp [Puerta principal] Sending command to activate live stream ON-DEMAND
2022-08-29T21:00:34.661Z ring-mqtt [Puerta principal] Received set live stream state ON-DEMAND
2022-08-29T21:00:44.614Z ring-rtsp INF [path 343ea4da34a8_live] runOnDemand command stopped
2022-08-29T21:00:44.615Z ring-rtsp INF [RTSP] [conn 10.0.0.2:35240] closed (source of path '343ea4da34a8_live' has timed out)
2022-08-29T21:00:44.615Z ring-rtsp [Puerta principal] Deactivating live stream due to signal from RTSP server (no more active clients or publisher ended stream)
2022-08-29T21:00:44.635Z ring-mqtt [Puerta principal] Received set live stream state OFF
2022-08-29T21:00:44.638Z ring-mqtt [Puerta principal] Stopping live stream session on workerId 1
2022-08-29T21:00:47.970Z ring-mqtt [Puerta principal] ring/ec5e4717-523d-43b6-81b2-14fa452f0e15/camera/343ea4da34a8/status offline
2022-08-29T21:00:49.731Z ring-rtsp INF [RTSP] [conn 10.0.0.2:42136] opened
2022-08-29T21:00:49.748Z ring-rtsp INF [path 343ea4da34a8_live] runOnDemand command started
2022-08-29T21:00:49.762Z ring-rtsp [Puerta principal] Sending command to activate live stream ON-DEMAND
2022-08-29T21:00:49.808Z ring-mqtt [Puerta principal] Received set live stream state ON-DEMAND
2022-08-29T21:00:59.749Z ring-rtsp INF [path 343ea4da34a8_live] runOnDemand command stopped
2022-08-29T21:00:59.750Z ring-rtsp [Puerta principal] Deactivating live stream due to signal from RTSP server (no more active clients or publisher ended stream)
2022-08-29T21:00:59.750Z ring-rtsp INF [RTSP] [conn 10.0.0.2:42136] closed (source of path '343ea4da34a8_live' has timed out)
2022-08-29T21:00:59.769Z ring-mqtt [Puerta principal] Received set live stream state OFF
2022-08-29T21:00:59.769Z ring-mqtt [Puerta principal] Stopping live stream session on workerId 1
2022-08-29T21:01:37.974Z ring-mqtt [Puerta principal] ring/ec5e4717-523d-43b6-81b2-14fa452f0e15/camera/343ea4da34a8/status online
2022-08-29T21:02:25.842Z ring-attr [Puerta principal] ring/ec5e4717-523d-43b6-81b2-14fa452f0e15/camera/343ea4da34a8/info/state {"firmwareStatus":"Up to Date","lastUpdate":"2022-08-29T21:00:30Z","wirelessNetwork":"FLEXA_5E9B_outdoor","wirelessSignal":-70,"stream_Source":"rtsp://****:****@172.16.0.12:8554/343ea4da34a8_live","still_Image_URL":"https://localhost:8123{{ states.camera.puerta_principal_snapshot.attributes.entity_picture }}"}
2022-08-29T21:02:25.843Z ring-attr [Puerta principal] ring/ec5e4717-523d-43b6-81b2-14fa452f0e15/camera/343ea4da34a8/wireless/attributes {"wirelessNetwork":"FLEXA_5E9B_outdoor","wirelessSignal":-70}
^[[1;2A2022-08-29T21:03:08.650Z ring-mqtt [Puerta principal] All motion dings for camera have expired
2022-08-29T21:03:08.650Z ring-mqtt [Puerta principal] ring/ec5e4717-523d-43b6-81b2-14fa452f0e15/camera/343ea4da34a8/motion/state OFF
2022-08-29T21:03:08.651Z ring-attr [Puerta principal] ring/ec5e4717-523d-43b6-81b2-14fa452f0e15/camera/343ea4da34a8/motion/attributes {"lastMotion":1661806807,"lastMotionTime":"2022-08-29T21:00:07Z","personDetected":true,"motionDetectionEnabled":true}
2022-08-29T21:04:40.013Z ring-mqtt [Puerta principal] ring/ec5e4717-523d-43b6-81b2-14fa452f0e15/camera/343ea4da34a8/status offline
2022-08-29T21:05:10.021Z ring-mqtt [Puerta principal] ring/ec5e4717-523d-43b6-81b2-14fa452f0e15/camera/343ea4da34a8/status online

tsightler commented 2 years ago

Are you willing to share the camera?

DiEVeXx commented 2 years ago

Yes, i can, can you explain me about this?

EDIT: Email sent

tsightler commented 2 years ago

Thanks for be willing to share the camera, it makes testing so much easier. I added the camera to my development setup and everything works perfectly so that at least tells me that there's nothing really wrong with ring-mqtt itself, or at least nothing that should keep it from working, so it's more likely to be something specific to your environment.

It's a little difficult to troubleshoot because the logs above don't include all details and it looks like this wasn't the very first attempt. It looks a little like the first attempt failed for some reason and then it got into an unexpected state because it's never really trying to start a stream in those logs. Can you please run with DEBUG=ring-* and then try again and provide the full logs? You can zip them up and email them to me if that is easier for you.

I'm going to remove your camera from my account since we've been able to prove that it at least should work and I know it's a little weird to share a camera with a random person on the internet!

DiEVeXx commented 2 years ago

Logs sent. Thanks again.

tsightler commented 2 years ago

Hi @DiEVeXx, thanks for the logs. There is something very strange about them that I also noticed in the logs above. Again, ignoring the streaming issue there appears to be a very unstable network connection between the host running ring-mqtt and the Ring API servers, there are tons of request failures and polling fails so often that the device goes online/offline over and over (polling must fail 3 times in 60 seconds for this to happen). Lots of requests fail randomly and work later.

Based on this, it seems like the device you are running ring-mqtt on has very poor network connectivity and of course live streaming will never work this way. In one of the log cases the stream did actually start, but it took more than 8 seconds. For example, with my own RPi4 starting a stream to your camera takes about 3 seconds.

I'm not really sure what might be causing this, are you using Wifi or Ethernet? Do you have other things running on the host? It could possibly be MTU related, I thought that because it seemed like bigger queries (requests for event list or for snapshots) seem to fail much more often than small queries.

In the end, until you can eliminate these errors, there's not much reason to attempt to troubleshoot the streaming issues because of course streaming won't work if connection to the Ring API is not stable.

DiEVeXx commented 2 years ago

The ring-mqtt service is running on a Raspberry 4 4GB connected by cable to a switch, it is running the services mentioned in the yml and some more.

Tomorrow I will try to connect it directly to the router and try to reduce the amount of services running (maybe the switch is overloaded), I will also take a look at the MTU issue. Thanks for your help

tsightler commented 2 years ago

OK, well let me know what you find. The logs are full of disconnects and errors retrieving data from URLs. I have no idea why this would be the case. I suppose it could also be RAM or CPU resources, but that doesn't appear to be an overloaded system from that perspective.

Out of curiosity, what version of Docker are you running and, by any chance are you running a 32-bit host OS?

DiEVeXx commented 2 years ago

Docker version 20.10.5+dfsg1, build 55c4c88 Yes, i'm using 32bits OS version, is this a problem?

tsightler commented 2 years ago

Docker version 20.10.5+dfsg1, build 55c4c88 Yes, i'm using 32bits OS version, is this a problem?

Technically, using 32-bit OS shouldn't be an issue, but it's not something I test at all. I've switched all my RPi's to 64-bit Ubuntu or 64-bit Home Assistant OS some time ago. In the past there was an issue with recent Alpine versions and the older Docker shipped with Raspbian 10 that would cause various issues, including network connectivity issues. I thought these issues were addressed with Raspberry OS 11, but I wonder if there's still some less obvious issues that are causing the network behavior.

Honestly that's just a shot in the dark because there's nothing obvious here. I've seen IPv6 cause some users a few issues as well, but I doubt that's it. I also notice you are creating an overlay network rather than using the default bridge network, I would also consider this uncommon vs other users, and I believe overlay networks do impact MTU since I believe they encapsulate traffic inside VXLAN, so you might try setting the MTU to 1400 and seeing what happens.

It might also be interesting to connect to a bash shell in the container and see if you can reproduce the network issues from the command line. For example, install speedtest-cli (apk add speedtest-cli) and run that a few times over the course of a minute or two and see what it reports.

Just ideas, as I'm not really sure what else I can do at this point.

tsightler commented 2 years ago

One other possible issue, do you definitely have a caching DNS on your local network? Do you have any way to monitor DNS queries going out of your network? I remember an old bug that was never tracked down where sometimes ring-client-api would just start making 1000's of DNS queries/second and it could cause behavior similar to this. I was only able to reproduce the issue a handful of times, and only by forcing DNS to use an external DNS server directly. Perhaps a packet capture on the host running ring-mqtt would give some clue.

DiEVeXx commented 2 years ago

I have already found the problem. The problem really was in the network instability of the raspberry pi, apparently there is a bug if a static IP address is set which causes a lot of network packets to be lost, once a dynamic IP address is set and assigned in the DHCP server a static IP the problem has been solved.

Thank you very much for your support, I close the issue.

tsightler commented 2 years ago

Great, glad you got it figure out and thank you for sharing the solution! I do think I remember seeing that issue come up in my searches at some point.

tsightler / ring-mqtt