Push notifications stop after ~24 hours

[dgreif]

In #1212, we confirmed that clearing existing devices and re-authenticating seems to fix Push Notifications. However, some users are finding that the notifications only work for ~24 hours before they need to do the process again. The native apps don't seem to have this issue, so they must be sending some update to keep notifications working long term. We should investigate this further and find a way to avoid the daily bug.

See https://github.com/dgreif/ring/issues/1212#issuecomment-1546755943 for more details.

[salemsayed]

Correct, notifications worked for 1 or 2 days max then stopped also for my Ring Intercom

[smockle]

We should investigate this further…

If someone was interested in conducting this investigation, what steps would they need to take? Setting up mitmproxy and watching app traffic for a day or so?

[dgreif]

Great question @smockle! The hard part with mitmproxy is that the Ring App uses certificate pinning, which makes any MITM setup not work by default. There are ways around this, but they require a rooted device and a few pieces of software to get everything working. It's not a route I recommend others try to go down 😅.

Honestly I think this issue might be one that we can debug without necessarily seeing the traffic from the App. Based on the assessment from @tsightler in the other issue, I have a suspicion that the subscribed values are getting set to false after some period of time (24-48 hrs), so theoretically we should be able to take a token and hardware id belonging to a session from a few days ago and see what those values are. It might be as simple as watching those values and re-subscribing if they go false.

Another suspicion is that users experiencing this issue might be on a setup where they are not getting a consistent hardware id. This would present in the form of many "devices" showing up in the Control Center during that first 24 hour period. @salemsayed I'm curious, after you cleared your devices and waited a day, did you have just a few devices in Control Center, or was the list very long again?

[tsightler]

It might be as simple as watching those values and re-subscribing if they go false.

Unfortunately, the code in ring-mqtt already does this, at least it attempts to by calling subscribeToDingEvents()/subscribeToMotionEvents() when it detects this by checking the value on every poll event (20 seconds). The thing is, subscribing to the events doesn't appear to fail, but the observed behavior in the logs is that, once these values start showing up as false, they will revert back to unsubscribed every few minutes even after calling the subscribe function.

@smockle If you do decide to go down the path of mitmproxy, below is a post from a ring-mqtt user that did just that when they wanted me to implement a function that we didn't know how the API worked. It might help if you decide to go down that route, it didn't seem too hard.

https://github.com/tsightler/ring-mqtt/discussions/528#discussioncomment-4715702

I actually want to set this up myself, but time just hasn't been available for me lately.

[salemsayed]

Great question @smockle! The hard part with mitmproxy is that the Ring App uses certificate pinning, which makes any MITM setup not work by default. There are ways around this, but they require a rooted device and a few pieces of software to get everything @salemsayed I'm curious, after you cleared your devices and waited a day, did you have just a few devices in Control Center, or was the list very long again?

Great question @smockle! The hard part with mitmproxy is that the Ring App uses certificate pinning, which makes any MITM setup not work by default. There are ways around this, but they require a rooted device and a few pieces of software to get everything working. It's not a route I recommend others try to go down 😅.

Honestly I think this issue might be one that we can debug without necessarily seeing the traffic from the App. Based on the assessment from @tsightler in the other issue, I have a suspicion that the subscribed values are getting set to false after some period of time (24-48 hrs), so theoretically we should be able to take a token and hardware id belonging to a session from a few days ago and see what those values are. It might be as simple as watching those values and re-subscribing if they go false.

Another suspicion is that users experiencing this issue might be on a setup where they are not getting a consistent hardware id. This would present in the form of many "devices" showing up in the Control Center during that first 24 hour period. @salemsayed I'm curious, after you cleared your devices and waited a day, did you have just a few devices in Control Center, or was the list very long again?

Just two devices right now, an unnamed device and homebridge-ring device.

[tammotheus]

Just two devices right now, an unnamed device and homebridge-ring device.

Same here. Have been deleting and re-authenticating the devices for days now. Each time it's just the two: unnamed and Homebridge ring.

[daneedk]

Hi,

I use the ring-client-api in an app for the Homey Smart Home Platform (https://homey.app/) and there are quite some users of the app that are reporting the same exact problems with the notifications. The strange thing is that not all users seem to be affected by the bug, the app has ~5000 users but only a few compared to that amount have reported it. For some the removal of the devices at Ring has been a solution for a couple of days now, for others it only lasted a day. In my app the api connection is registered using the hardware ID of the Homey device, this is obviously unique per Homey and does not change. This results in 2 registrations at Ring, one with the name Homey (hardcoded in my app) and one with the name ring-client-api. When the app is restarted and thus a new connection is made and notifications are registered again there are no new devices created at Ring. Restarting the app does not necessarily cause the notifications to stop working, for some it’s a problem for others it isn’t. There are also users who lost notification that did not restart the app during the period it worked.

That’s all information I can add for now, I’m currently on vacation and have currently no way to do any troubleshooting myself. When I get more relevant information from my users I will post it here.

Edit: For anyone interrested, this is the code that sets up the connection with the Ring api: https://github.com/daneedk/com.amazon.ring/blob/22b0d0596e1114a374b9ef57757d88ac971cbb0e/lib/Api.js#L97

[dgreif]

Thanks for the info @daneedk! That sounds in-line with the other experiences described so far.

I personally use the standard Ring App notifications so I wasn't sure if this was affecting me. Enabled notifications this morning and mine are working just fine, so I'm not able to repro/debug 😞.

It sounds like there is a lot of variance in who is experiencing this. For those having issues, what version of Node.js are you on? I'm on 18.16.0 without issues.

[tsightler]

@daneedk That sounds exactly the same as what I'm seeing from ring-mqtt users as well. I estimate at least 6000 users of ring-mqtt based on various analytics data (it's probably higher, but so far only a few sporadic reports. Almost all started around May 4th and it's the first time it ever impacted my setup. However, since resetting all authorized users, my setup works perfectly for two weeks now.

However, its clear that other users see this work only for about 24 hours. For whatever reason, I've noticed that a lot of people with the issue do not have a Ring subscription, but not everyone. Perhaps there's just a higher percentage of Ring users that don't have a subscription than I realized. I'm also not sure that anyone that has reported the issue to this point is a US resident, so I wonder if the problem is happening in other regions vs the US.

However, I think it would be really good if we could start collecting this basic information from users experiencing the 24 hour issue to see if there's anything common. My suggestions for questions:

1) When did you notice the problem? 2) Did you remove all device and re-authenticate, if so, did this work and for how long? 3) What devices do you have? 4) What region/country are you located in? 5) Do you have a ring subscription?

However, I'm also open to other suggestions. Not being able to reproduce the issue makes it particularly difficult to find any solution. One thing, at least one user has reported that the native Home Assistant Ring integration has the same problem, but it still uses the old polling method vs push notifications (as far as I know anyway).

[daneedk]

This problem occurs on both the old and new Homey. The old Homey uses nodejs v12.22.11 and the new Homey, which is really new and it’s firmware is still in Release Candidate phase, uses v18.16.0.

The old Homey has not been available in the US so no US based users for that one, the new Homey will be available in the US, but I’m not sure if any are shipped yet (I’ll try to get that confirmed)

The first report about this particular problem was posted on the Homey Community forum on the 10th of may. My app has a dedicated support topic on that forum, I will ask users that are experiencing the problem to answer the questions above.

[tammotheus]

1. When did you notice the problem? First week of May
2. Did you remove all device and re-authenticate, if so, did this work and for how long? Six times so far. Between 16 and 36 hours.
3. What devices do you have? Ring Intercom, Ring Chime
4. What region/country are you located in? Germany
5. Do you have a ring subscription? No

[pepepere]

1. When did you notice the problem? First week of May
2. Did you remove all device and re-authenticate, if so, did this work and for how long? Yes. Between 16 and 24 hours with ding events
3. What devices do you have? Ring Doorbells (2).
4. What region/country are you located in? Europe/Spain
5. Do you have a ring subscription? No

[CooperCGN]

1. When did you notice the problem? First week of May
2. Did you remove all device and re-authenticate, if so, did this work and for how long? Yes, worked for around 48 hours, maybe a coincidence but it stopped after I had to restart Homebridge
3. What devices do you have? Ring intercom
4. What region/country are you located in? Germany
5. Do you have a ring subscription? No

[dgreif]

Thanks for the responses everyone!

However, its clear that other users see this work only for about 24 hours. For whatever reason, I've noticed that a lot of people with the issue do not have a Ring subscription, but not everyone. Perhaps there's just a higher percentage of Ring users that don't have a subscription than I realized. I'm also not sure that anyone that has reported the issue to this point is a US resident, so I wonder if the problem is happening in other regions vs the US.

@tsightler you may have nailed the problem in this paragraph. I think the symptoms we are seeing likely stem from new Data Residency laws, which require data to only be stored within the region of the user. In some of the new laws, there is an acceptable period for data to be stored in another region, but only temporarily (something on the scale of 24-48 hours). If Ring is storing the session and push notification data in the US, it would make sense that they would only be able to store that same info for 24-48 hours for users in Spain/Germany.

Assuming this theory is correct, I'm guessing the apps are now configured to recreate the user session and push notification registration at least once a day. We should be able to do something similar from ring-client-api, re-posting these on a periodic basis. I haven't had a chance to try it out yet, but I think this should be pretty simple to change. I'll post back here as soon as I have some progress. I have a pretty relaxed weekend so I might be able to get to it tomorrow 🤞

[AAMM007]

Hi, I’m having the same problem here in Canada.

When did you notice the problem? A few weeks ago, can’t tell exactly.

Did you remove all device and re-authenticate, if so, did this work and for how long? yes, 1-2 days

What devices do you have? Doorbell pro and stickup cam

What region/country are you located in? Canada

Do you have a ring subscription? Yes

[tsightler]

@dgreif I think the theory of changes regarding data residency may make sense. Around the same time reports started of notification issues there was also another issue reported with ring-mqtt that I was pretty sure was indirectly related to the infrastructure changes. The ring-mqtt code polls the device health API every 5 minutes to update various attribute data. Around the same time notifications started having issues some users also reported an issue where this data was generating an error in ring-mqtt due to null data.

Investigating this issue, it appeared to only impact EU users. Querying the API still returned a result, but the JSON response mostly contained null values, especially for personally identifiable data like wifi network name. After about 24-48 hours everyone that reported the problem indicated that the behavior returned to normal. I had thought that maybe Ring was making changes to keep personally identifiable data, like wifi names, in-region.

Assuming this theory is correct, I'm guessing the apps are now configured to recreate the user session and push notification registration at least once a day.

One thing though, does this not happen on restart? So far users have reported restarting doesn't help.

[daneedk]

@dgreif @tsightler the theory about it impacting mostly EU users sounds plausible, most users of my app are in the European region. There are also some in Australia and New Zealand, I can contact a user from Australia and check with him. Btw, here’s my info, I’m collecting the rest of the information from other users and will post that later.

1. When did you notice the problem? Around 12 may
2. Did you remove all device and re-authenticate, if so, did this work and for how long? Yes, on three different Homeys, it worked between 24-48 hours
3. What devices do you have? Ring Video Doorbell Pro 2, Ring Video Doorbell Pro, Stickup Cam, Indoor Cam
4. What region/country are you located in? EU, the Netherlands
5. Do you have a ring subscription? Yes

[jchristianj]

Having the issue here as well. No dings and no motion events coming through to Homebridge. Ring App works. Node.js version is 18.16.0

1. When did you notice the problem? Beginning of May

2. Did you remove all device and re-authenticate, if so, did this work and for how long? Yes, worked for max 24 hours.

3. What devices do you have? Ring Pro 2 wired and Ring 2 battery doorbells

4. What region/country are you located in? EU, Germany

5. Do you have a ring subscription? Yes

[dgreif]

I've released v11.7.6 which recreates the user session every 12 hours, and attaches the most recent push notification token whenever the session is recreated. I'd appreciate help testing this since I can't verify it within the US! Anyone who updates, please let me know if notifications stop working around the 24 hour mark still, otherwise hopefully we will know this is working in 2-3 days 🤞

One thing though, does this not happen on restart? So far users have reported restarting doesn't help.

Great question! The original setup actually didn't create a session (even on restart) unless we got a 404 back on a standard data request. Now I'm creating a session before any data requests go out, as well as on a 12 hour interval.

[tsightler]

For users of ring-mqtt that may be monitoring this issue, I've updated the dev branch to include 11.7.6 of ring-client-api to make this easier to test. Simply change the "branch" config option from "addon" to "dev" and restart the addon/docker container and it will pull down the latest dev branch for testing purposes. If this works I'll push out a full update later this week.

[MoTechnicalities]

Well, I'm in the U.S., Michigan, and I have a Ring subscription. I've changed nothing since the Motion Detection on the cameras stopped getting to Homebridge.

Except for now I've updated the plugin. There is still no change. Cameras still no longer pass Motion Detection to Homebridge. ☹️

None of these cameras are passing motion:

[tsightler]

@MoTechnicalities Did you follow the instructions of deleting all authorized clients and re-authenticating.

[benjackson1111]

The issue I was bringing attention to here: https://github.com/dgreif/ring/issues/1219 got closed as a duplicate of this post.

Just letting you know that the issue 1219 is not resolved after upgrading to the latest version 11.7.6, the button on the ringdoorbell is still not pushing any notifications to homekit

Thought you might like to know

[tsightler]

@benjackson1111 Same question, have you followed the steps to remove all devices and re-authenticate? Does this fix it for some period of time?

[benjackson1111]

Sorry - yes that did it. Will report back if it stops again. Thank you!!

[MoTechnicalities]

👍 I deleted the Authenticated Devices and the Motion Sensors are back on line.😀

Will keep you posted. Thanks.

[daneedk]

Thanks for the new version @dgreif. Unfortunately I can’t test it yet as I’m still on vacation and won’t be home until saturday.

[salemsayed]

I'm not in the US, updated to latest version and will report back after 24H

[taher51]

I am in the UK, updated to the latest version, and will report back in 24H.

[McJohnPi]

@dgreif Thank you so much for your nice and fast work!

We have had the same problem here in the Netherlands. Just did the update to v11.7.6. But still the same results; no ring/push and motion notifications. Thought that we had deleted all the existing devices but that was in the Ring master account... Then we deleted all the devices from the separate homebridge ring (Shared access) account, made a new token and voila! There was Homekit sound and motion again! (The strange thing was that the Homekit live view was working well all the time). We look forward to tomorrow to hear if it still works... Thanks again for all your work, family is happy again! 👍

[ultrathew]

I am in the US, have updated to the newest version of the plugin, and removed and re-added my Ring doorbell and I'm still not getting motion or doorbell ring alerts at all.

I'm also now getting this in my Homebridge logs:

[5/22/2023, 12:12:41 PM] [homebridge-ring] Failed to connect push notification receiver [5/22/2023, 12:12:41 PM] [homebridge-ring] Error: GCM register has failed at postRegister (/usr/local/lib/node_modules/homebridge-ring/node_modules/@eneris/push-receiver/src/gcm.ts:87:19) at processTicksAndRejections (node:internal/process/task_queues:95:5) at doRegister (/usr/local/lib/node_modules/homebridge-ring/node_modules/@eneris/push-receiver/src/gcm.ts:54:22) at exports.default (/usr/local/lib/node_modules/homebridge-ring/node_modules/@eneris/push-receiver/src/gcm.ts:15:25) at PushReceiver.register (/usr/local/lib/node_modules/homebridge-ring/node_modules/@eneris/push-receiver/src/client.ts:169:30) at PushReceiver.connect (/usr/local/lib/node_modules/homebridge-ring/node_modules/@eneris/push-receiver/src/client.ts:122:36) at RingApi.registerPushReceiver (/usr/local/lib/node_modules/homebridge-ring/node_modules/ring-client-api/lib/api.js:179:13)

[djm181]

Hmm, this only works for me the first time I restart after deleting the authorized device. If I recycle my client again then it stops receiving push notifications again. Deleting the client device fixes the problem again.

[McJohnPi]

@djm181 you are right. Unfortunately same problem here; after a restart of Homebridge it stops receiving push notifications again...

[pepepere]

Sorry, unfortunately the problem persits with ring-mqtt / HA in docker with branch option to dev I think it has been running for 18 hours, until the system was rebooted. Thereafter, neither ding nor motion events

[jchristianj]

can confirm that v11.7.6. notifications survive 24 hours, but are killed by home bridge restart.

[saz969]

I live in the US and have the same issue reported by other users, so the reference to the EU may be a red herring:

1. When did you notice the problem? Earlier in May

2. Did you remove all device and re-authenticate, if so, did this work and for how long? Yes, a number of times, it works for a period of between 24-48 hours.Also setup a a dedicated account to see if that helps, but same issue occurs.

3. What devices do you have? Ring Video Doorbell Pro 2, Ring Video Doorbell Pro, Ring Floodlight Cam Wired Pros x 2

4. What region/country are you located in? USA

5. Do you have a ring subscription? Yes

While this may not be relevant, I wanted to make sure that everyone was aware of this notice from Ring that is an indication of some changes taking place in their backend.

https://support.ring.com/hc/en-us/articles/360059615351-Ring-Desktop-App-will-be-Discontinued-in-2023

[MoTechnicalities]

Tonight I Restarted my entire Raspberry PI Server just to test things out. Here is the order: I do not run the Ring plugin in a Child Bridge. It's been running since my last post up above. 1) It worked great until this [Restart Server] but returned to NO Motion Detections after the restart. 2) I deleted the Authenticated Unit (Android-Homebridge) from the iPhone Ring App and Refreshed the Homebridge Token. Still no detections! 3) I then did a [Quick Restart] (NOT a full [Restart Server]) and Motion Detection came back.

For other testers: I did not repeat the process a second time, but since it came back after the [Quick Restart] that followed the Token Refresh, It may be only goofing up after a full [Restart Server]. I'm not sure yet and will wait.

[McJohnPi]

I was wondering why the Ring plugin causes 2 new devices in the Ring Authorized device environment?

[dgreif]

That's a fair question @McJohnPi. I'll track that down separately

[eightball011854]

i think i may have identified the issue

the motion sensors tied to the ring cams get stuck in an Updating status within homekit, which then stops notifications from coming through

any possible fix for this, if it is in fact the same issue

When did you notice the problem? Earlier in May

Did you remove all device and re-authenticate, if so, did this work and for how long? Yes, a number of times, it works for a period of time of a few hours

What devices do you have? Ring Video Doorbell Pro, Ring Floodlight Cam Wired Pro

What region/country are you located in? USA

Do you have a ring subscription? Yes

[rodjager]

Firstly, @dgreif thank you for your continued efforts.

The refresh steps work for restoring notifications but only for around 24 hours.

As noted by others, I also see 2 authorised devices via Ring's control panel after reregistering Homebridge.

After that 24 hour period I start to receive a series of errors in the Homebridge log similar to this: [5/25/2023, 9:11:25 PM] [Ring] Failed to reach Ring server at https://app.ring.com/api/v1/mode/location/. Timeout awaiting 'request' for 20000ms. Trying again in 5 seconds...

Whatever causes this timeout also seems to result in the network being blocked from connecting to anything ring.com related as all connections (apps and hardware) report being offline. To reset the IP block I need to disable the Ring plugin in Homebridge and wait between 30-60 minutes.

To answer the various questions:

1. When did you notice the problem? Earlier in May

2. Did you remove all device and re-authenticate, if so, did this work and for how long? Yes, around 24 hours each time. Have been testing over the last few days and seeing this consistently.

3. What devices do you have? Ring Video Doorbell (2nd edition), Ring chime

4. What region/country are you located in? Australia

5. Do you have a ring subscription? No

[McJohnPi]

The strange thing is that in our case the push notifications are working perfect after more than 48 hours. But we are punished horribly if we (accidentally) restart HomeBridge.. So, that's now forbidden.

[benjackson1111]

In my case, I automatically have HB restart every 24 hours for stability. So yes, I can confirm as well that it worked perfectly before a restart.

[JeffGoldner]

When did you notice the problem? Not sure

Did you remove all device and re-authenticate, if so, did this work and for how long? Yes, but it doesn't really work. The first notification works, then it stops.

What devices do you have? Ring Video Doorbell (2nd edition), Ring chime

What region/country are you located in? US

Do you have a ring subscription? No

[JeffGoldner]

I have noticed that when I get a notification through scrypted, the Ring app shows that as "Answered Motion". Following that, subsequent motions show up as "Motion" in the event log, not "Answered Motion" - and I don't get Ring app notifications for these nor does anything get through to the Scrypted plug-in. And this seems to be the pattern each day: First motion of the day, I get "Answered Motion", Scrypted catches it, throws it to Homekit, life is as expected. Then nothing for the rest of the day as all subsequent motion events are showing "Motion". I believe this was also the case a few days ago when I was just using the homebridge-ring plug-in.

I have cleared all the crud from previous apps, just two authorized now, so they aren't piling up unless I do a new login, then I get two more (Unnamed app, ring-api).

[JeffGoldner]

And it could very well be the case that restarting Scrypted allows the first notification, but that doesn't explain Ring app behavior.

[tsightler]

@JeffGoldner Not an expert on Scrypted, but I'm pretty sure it uses a forked version of ring-client-api that re-implements polling. Also, there was a new version of the Scrypted Ring plugin pushed just two days ago that attempts to fix the polling interface for the Ring plugin in Scrypted. Are you using that version?

I'd be quite surprised if polling doesn't work, but I've seen other reports of this with the Home Assistant Ring addon as well, so I'm starting to be suspicious.

[JeffGoldner]

Already discussing with Koush. That update did one thing - it terminated motion after 30 seconds. That part does work, but otherwise it's subject to the same thing. He's watching this as well I believe.

[tsightler]

OK, but my point is that, unlike code built directly on ring-client-api here, that code uses both push notifications and polling as it subscribes to both onMotionDetected and onMotionDetectedPolling, the latter being included only in the Scrypted fork of ring-client-api. This implies that the problem might not be directly related to push notifications, but more broadly even with polling for dings, which is a little bit surprising for me.

[tsightler]

@dgreif I don't know if this will help you in any way, but I thought I would share it.

I am basically also seeing the behavior reported above, i.e. if I remove the account and re-authenticate, then notifications will work for the initial connection, and seem to survive mulitple days (worked from Sunday of last week until Saturday). However, if I restart ring-mqtt which uses the existing saved token, then push notifications will not work (well, they do seem to randomly work maybe one time out of 10, but mostly they do not work).

However, what I've discovered is that I don't have to completely re-authenticate to get them to work again. As noted above there are two "Authorized Client Devices" that show up with ring-client-api, one that has "Device name not found" and the other with whatever name the client uses ("ring-mqtt" or "ring-mqtt-addon" in the case of my project). This has of course been the case since the start of the project, and I've never really concerned myself with it, but I thought it was interesting.

What I've found is that, if I delete the authorized device with the registered name, but do not delete the one with "Device name not found", then I can restart ring-mqtt and authentication will still work with the token, and the second authorized client will be registered again with the name, but notifications will start working again for that session.