Closed ramonbgc closed 3 weeks ago
Hey there @allenporter, mind taking a look at this issue as it has been labeled with an integration (nest
) you are listed as a code owner for? Thanks!
(message by CodeOwnersMention)
nest documentation nest source (message by IssueLinks)
Ok this is odd it shouldn't need to create anything if its already set up. I will have a closer look. Edit: Nevermind, creating a subscriber is normal. (Creating the subscription is only done on setup)
Can you confirm your pubsub topic is enabled in the device access console?
Are you able to getdiagnostics information and attach it? https://www.home-assistant.io/docs/configuration/troubleshooting/#download-diagnostics
Experience the same error after working for a long time
Can you confirm your pubsub topic is enabled in the device access console?
Yes, it's enabled:
Are you able to getdiagnostics information and attach it?
I don't see that option (don't know why):
but I've uploaded the "debug logging" file if that helps. home-assistant_nest_2024-07-31T07-14-58.678Z.log
I'm getting the same error, my Pub/Sub topic was disabled, but I re-enabled it. Still getting the error.
Updated my IPv6 to Automatic and the integration started working again.
IPv6
Where have you enabled it? I'm using a docker container and I don't see that kind of option on the HA configuration.
I set this up on my Raspberry Pi under system and then network
Thanks @Arjan477
I'm using a docker container and that option is not available on Home Assistant when a container is used. I've tried to enable IPv6 on the container but it didn't work, I had the same issue.
@allenporter, any test you want me to do? any way from my side to check what is happening? THANKS!!
@ramonbgc check for any issues with ipv6 on your network or any ad blocking software that may be blocking google. These are the most common two reasons for timeouts.
I don't use and didn't ever use IPv6 on my network (I haven't made any change on the router in the past months) Regarding the blocking software, I've been using Pihole for a long time, and I'm still using it, but I already tried disabling it and the integration didn't work neither 😥
Any other suggestion?
I tried enabling IPv6 and it didn't do anything. I removed the nest and added it back. That worked for me.
I've already done the following tests:
None of them worked for me 😥
@allenporter, is there any , not so complicated, way to "debug" the code? thanks!
I believe we have the information from the error messages: it's a timeout from what I can see. (Enabling debug and diagnostics give detail as well butay not add much than we can see)
I believe I have the exact same issue.
System is official HA docker running in Unraid.
Nest integration was running ok ~4 weeks ago (didn't remember particular version at that time) but suddenly stopped worked with error msg:
Subscriber error: Failed to create subscriber 'projects/[REDACTED]/subscriptions/home-assistant-[REDACTED]':
Have verified that pub/sub API is/has been "Enabled".
Debug log is almost the same as OP with Creating subscriber
then credentials refreshed
.
Nothing else interesting or obviously related.
There's no network level filtering/dns that are filtering on top of HA.
Thanks for help.
Can you try with home assistant in safe mode? What is the behavior like on startup?
Having additional debug log captures would help. I realize it may look the same from the error messages, but the timing and ordering of messages is helpful. (The above trace shows logs from a pubsub subscriber working for a long time, but then it appears its performance is trashed causing threads to take a long time)
@allenporter I'll jump in with some safe mode results:
I had the Nest integration disabled before I restarted in Safe Mode, this screenshot lines up with about 5 seconds before "[hass_nabucasa.google_report_state] Connected" and then repeats roughly in line with the timeouts. This will also cause python3 to eventually consume all memory available until the system kills it, so I've been leaving it disabled.
2024-08-12 16:19:34.063 INFO (MainThread) [homeassistant.setup] Setting up nest 2024-08-12 16:19:34.413 DEBUG (SyncWorker_2) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' 2024-08-12 16:19:34.712 DEBUG (SyncWorker_2) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed 2024-08-12 16:19:59.930 INFO (MainThread) [hass_nabucasa.google_report_state] Connected 2024-08-12 16:20:09.747 DEBUG (SyncWorker_12) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' 2024-08-12 16:20:10.044 DEBUG (SyncWorker_12) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed 2024-08-12 16:20:50.093 DEBUG (SyncWorker_6) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' 2024-08-12 16:20:50.404 DEBUG (SyncWorker_6) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed 2024-08-12 16:20:53.277 DEBUG (SyncWorker_2) [google_nest_sdm.google_nest_subscriber] Subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' configured on topic 'projects/sdm-prod/topics/enterprise-REDACTED' 2024-08-12 16:20:53.277 DEBUG (SyncWorker_2) [google_nest_sdm.google_nest_subscriber] Starting subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' 2024-08-12 16:20:55.089 INFO (MainThread) [hass_nabucasa.google_report_state] Timeout while waiting to receive message 2024-08-12 16:21:28.548 DEBUG (SyncWorker_12) [google_nest_sdm.google_nest_subscriber] Subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' configured on topic 'projects/sdm-prod/topics/enterprise-REDACTED' 2024-08-12 16:21:28.548 DEBUG (SyncWorker_12) [google_nest_sdm.google_nest_subscriber] Starting subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' 2024-08-12 16:21:40.164 DEBUG (SyncWorker_2) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' 2024-08-12 16:21:40.456 DEBUG (SyncWorker_2) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed 2024-08-12 16:21:50.157 INFO (MainThread) [hass_nabucasa.google_report_state] Timeout while waiting to receive message 2024-08-12 16:22:10.107 DEBUG (SyncWorker_6) [google_nest_sdm.google_nest_subscriber] Subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' configured on topic 'projects/sdm-prod/topics/enterprise-REDACTED' 2024-08-12 16:22:10.107 DEBUG (SyncWorker_6) [google_nest_sdm.google_nest_subscriber] Starting subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' 2024-08-12 16:22:45.229 INFO (MainThread) [hass_nabucasa.google_report_state] Timeout while waiting to receive message 2024-08-12 16:22:50.542 DEBUG (SyncWorker_3) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' 2024-08-12 16:22:50.836 DEBUG (SyncWorker_3) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed 2024-08-12 16:22:58.042 DEBUG (SyncWorker_2) [google_nest_sdm.google_nest_subscriber] Subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' configured on topic 'projects/sdm-prod/topics/enterprise-REDACTED' 2024-08-12 16:22:58.042 DEBUG (SyncWorker_2) [google_nest_sdm.google_nest_subscriber] Starting subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' 2024-08-12 16:23:40.298 INFO (MainThread) [hass_nabucasa.google_report_state] Timeout while waiting to receive message 2024-08-12 16:24:08.404 DEBUG (SyncWorker_3) [google_nest_sdm.google_nest_subscriber] Subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED' configured on topic 'projects/sdm-prod/topics/enterprise-REDACTED' 2024-08-12 16:24:08.404 DEBUG (SyncWorker_3) [google_nest_sdm.google_nest_subscriber] Starting subscriber 'projects/REDACTED-project/subscriptions/home-assistant-REDACTED'
I'm attempting to add additional timeouts and debug logs for a patch release to see if we can get additional information that will help narrow down the issue. Thanks everyone for your help in diagnosing this.
I've taken a Wireshark network trace while enabling the Nest integration and I haven't seen any timeout ... I'll try to do it again this evening and double check it.
I just started getting this error this evening - after upgrading to 2024.8.2.
New error code : 504 Deadline Exceeded. Log is the same as above I think
2024-08-16 22:24:51.308 DEBUG (MainThread) [google_nest_sdm.google_nest_subscriber] Starting subscriber projects/PROJ-NAME/subscriptions/home-assistant-PROJ-KEY
2024-08-16 22:24:51.321 DEBUG (SyncWorker_12) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/PROJ-NAME/subscriptions/home-assistant-PROJ-KEY'
2024-08-16 22:25:14.917 DEBUG (SyncWorker_12) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed
2024-08-16 22:25:40.315 DEBUG (MainThread) [google_nest_sdm.google_nest_subscriber] Starting subscriber projects/PROJ-NAME/subscriptions/home-assistant-PROJ-KEY
2024-08-16 22:25:40.333 DEBUG (SyncWorker_24) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/PROJ-NAME/subscriptions/home-assistant-PROJ-KEY'
2024-08-16 22:25:52.745 DEBUG (SyncWorker_24) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed
2024-08-16 22:26:13.194 DEBUG (MainThread) [google_nest_sdm.google_nest_subscriber] Starting subscriber projects/PROJ-NAME/subscriptions/home-assistant-PROJ-KEY
2024-08-16 22:26:13.196 DEBUG (SyncWorker_20) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/PROJ-NAME/subscriptions/home-assistant-PROJ-KEY'
2024-08-16 22:26:25.567 DEBUG (SyncWorker_20) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed
2024-08-16 22:26:55.881 DEBUG (MainThread) [google_nest_sdm.google_nest_subscriber] Starting subscriber projects/PROJ-NAME/subscriptions/home-assistant-PROJ-KEY
2024-08-16 22:26:55.883 DEBUG (SyncWorker_52) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/PROJ-NAME/subscriptions/home-assistant-PROJ-KEY'
2024-08-16 22:27:08.265 DEBUG (SyncWorker_52) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed
2024-08-16 22:27:58.514 DEBUG (MainThread) [google_nest_sdm.google_nest_subscriber] Starting subscriber projects/PROJ-NAME/subscriptions/home-assistant-PROJ-KEY
2024-08-16 22:27:58.516 DEBUG (SyncWorker_36) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/PROJ-NAME/subscriptions/home-assistant-PROJ-KEY'
For me, I haven't seen this prior to 2024.8.2, but I guess that could be co-incidence possibly.
I have the same issue as peteS-UK after upgrading to 2024.8.2, after downgrade to 2024.8.1 the issue goes away
following
Another one here. It was working for months, now it is not. Failed two days ago after the upgrade. Rollback did not help. NEST app on my phone works fine on local network.
Failed setup, will retry: Subscriber error: Failed to create subscriber 'projects/<REDACTED>/subscriptions/home-assistant-<REDACTED>' with api error: 504 Deadline Exceeded
Thanks we have the data needed to fix that issue and next patc release will have a fix
So this issue now has many reports of time deadline exceeded issue from the attempt to fix issues that was tracked in #124096. Once that fix lands in 2024.8.3 we'll want to see if the overall increased timeouts help folks seeing this issue. The root cause is the same that the RPCs are taking much longer than expected, which is odd.
I've updated HA to the latest version 2024.8.3 and enabled the Nest integration and I'm having this error:
2024-08-27 18:35:17.163 DEBUG (MainThread) [google_nest_sdm.google_nest_subscriber] Starting subscriber projects/rgc-home-assistant/subscriptions/home-assistant-eesoFlazSX 2024-08-27 18:35:17.165 DEBUG (SyncWorker_3) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/rgc-home-assistant/subscriptions/home-assistant-eesoFlazSX' 2024-08-27 18:35:17.364 DEBUG (SyncWorker_3) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed 2024-08-27 18:35:47.383 DEBUG (MainThread) [google_nest_sdm.google_nest_subscriber] Failed to create subscriber 'projects/rgc-home-assistant/subscriptions/home-assistant-eesoFlazSX' with api error: 504 Deadline Exceeded 2024-08-27 18:35:57.726 DEBUG (MainThread) [google_nest_sdm.google_nest_subscriber] Starting subscriber projects/rgc-home-assistant/subscriptions/home-assistant-eesoFlazSX 2024-08-27 18:35:57.728 DEBUG (SyncWorker_2) [google_nest_sdm.google_nest_subscriber] Creating subscriber 'projects/rgc-home-assistant/subscriptions/home-assistant-eesoFlazSX' 2024-08-27 18:35:57.884 DEBUG (SyncWorker_2) [google_nest_sdm.google_nest_subscriber] Subscriber credentials refreshed 2024-08-27 18:36:27.890 DEBUG (MainThread) [google_nest_sdm.google_nest_subscriber] Failed to create subscriber 'projects/rgc-home-assistant/subscriptions/home-assistant-eesoFlazSX' with api error: 504 Deadline Exceeded
The error message is different but it seems to be the same issue as the one before.
Hey @ramonbgc ! I had a similar situation but after deleting the integration and creating it again it worked. You will notice a longer than usual setup time in the last step - this is where the updated timeout kicked in. You can reuse all your assets at Google Cloud (OAuth ID, project name, credentials, etc...)
EDIT: I've tried deleting and creating the integration again and it didn't work. It just failed with the same error I showed above (I l also restarted HA).
thanks @agarkauskas, I tried before to delete and recreate the integration several times but it didn't work. let me try again now that I've updated to the latest version just in case.
@ramonbgc thank you for checking and reporting back. We have a little progress, but not answers yet, but i think we have narrowed down a bit to the problematic call.
So what we know is that the call to get_subscription
is taking over 30 seconds long. What we don't know is if (a) the timeout is due to a problem connecting to google or (b) if the problem is on the google side and it takes longer than 30 seconds to return details about the subscription.
We can test (b) by going to https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions/get and on the right side there is a way to test the API command. You can enter the subscription name from the log above under "creating subscriber" into the subscription
request parameter and press Execute
and authorize it and see the result on the bottom. (I believe this will just confirm the google side is working, and it should be fast).
I suspect there is still a network issue happening here where for some reason your docker container is unable to talk to google. You may want to look at Settings > System > Network and see if adjusting any ipv6 settings helps.
Let me know if you do or don't find anything and happy to keep discussing.
I just wanted to say 2024.8.3 is working fine for me but I did update directly from a working version of 2024.8.1 to 2024.8.3. I had rolled back to 2024.8.1 after seeing issues with 2024.8.2.
Just to add more color: If your instance broke in 2024.8.2 only then we're talking about #124096 where it broke due to trying to fix this issue (the request took longer than 10 seconds but less than 30 seconds). We are now talking in this issue about the original timeout here that existed before 2024.8.2. The root cause is likely the same as this request shouldn't really take 10+ seconds to begin with anyway.
@ramonbgc thank you for checking and reporting back. We have a little progress, but not answers yet, but i think we have narrowed down a bit to the problematic call.
So what we know is that the call to
get_subscription
is taking over 30 seconds long. What we don't know is if (a) the timeout is due to a problem connecting to google or (b) if the problem is on the google side and it takes longer than 30 seconds to return details about the subscription.We can test (b) by going to https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions/get and on the right side there is a way to test the API command. You can enter the subscription name from the log above under "creating subscriber" into the
subscription
request parameter and pressExecute
and authorize it and see the result on the bottom. (I believe this will just confirm the google side is working, and it should be fast).I suspect there is still a network issue happening here where for some reason your docker container is unable to talk to google. You may want to look at Settings > System > Network and see if adjusting any ipv6 settings helps.
Let me know if you do or don't find anything and happy to keep discussing.
Hi @allenporter! Thanks for your help!
I've tried b) and it worked (as you've anticipated):
{ "name": "projects/rgc-home-assistant/subscriptions/home-assistant-2hKTspozR9", "topic": "projects/sdm-prod/topics/enterprise-d654e46d-8879-4eb1-8be4-05de0297ab17", "pushConfig": {}, "ackDeadlineSeconds": 10, "messageRetentionDuration": "900s", "expirationPolicy": { "ttl": "2678400s" }, "state": "ACTIVE" }
OK so for some reason your home assistant instance takes 30 seconds, but your browser can do it instantly. The common reasons I see for this are ipv6 misconfigurations or adblockers that are misconfigured/broad. I know you've looked into both of these, but I don't really have any more insights.
ok, let me think about a way to test the connectivity between the container and the google api.
I'll keep you updated. thanks for the help!
Ok, that's a bit weird/embarrassing, I decided to delete the container and create a new one with the previous configuration and it worked. I don't know exactly why this was happening, as the network configuration is exactly the same ... could be some kind of Qnap's docker service bug ... I don't know 😥. The point is that just updating the HA it didn't work as I did it this week.
Thanks @allenporter for all the help and sorry for bothering you with such a "stupid" issue.
Fantastic, glad to hear everything is working.
The problem
After quite a long using the Google Nest integration in Home Assistant (more than one year) the integration stopped working showing the following error:
I've tried to recreate the credentials, grant permissions to the account again, recreate the pubsub subscription and even create a new google cloud project and went through all the steps ... without success 😥
The log shows that the subscriber is being created, the credentials refreshed and at least one event was retrieved:
but the integration initialization finally fails with the previous mentioned error (screenshoot)
I've been looking for similar issues and even when I've found some Nest integration issues this month (https://github.com/home-assistant/core/issues/122856) it's not exactly the same one I guess, as my subscription is correctly created by HA:
but the issue seems to happen after creation.
Thanks for your help!
What version of Home Assistant Core has the issue?
core-2024.7.4
What was the last working version of Home Assistant Core?
core-2024.6.1 (not sure)
What type of installation are you running?
Home Assistant Container
Integration causing the issue
Google Nest
Link to integration documentation on our website
https://www.home-assistant.io/integrations/nest
Diagnostics information
No response
Example YAML snippet
No response
Anything in the logs that might be useful for us?
No response
Additional information
No response