allenporter
commented
11 months ago (HVAC modes can no longer be corrupted by invalid pubsub messages. we now grab it on startup from the API then it never chagnes)
Closed ianwood closed 2 months ago
allenporter
commented
11 months ago (HVAC modes can no longer be corrupted by invalid pubsub messages. we now grab it on startup from the API then it never chagnes)
dredjohn21
commented
11 months ago Yes, i new error logs would be helpful. I am surprised its the same given that code was rewritten.
Thanks for helping to resolve. I'll get the logs and share. Maybe it's a different issue leading to same experience. My automation is simplistic: Looking for when my thermostat goes to OFF state. That was the easiest way to previously detect the problem and resolve. We'll see what is leading to that soon...
dredjohn21
commented
11 months ago (HVAC modes can no longer be corrupted by invalid pubsub messages. we now grab it on startup from the API then it never chagnes)
@allenporter it just happened again:
2023-10-18 07:00:02.526 INFO (Thread-5 (_run)) [google.api_core.bidi] Re-established stream
2023-10-18 07:00:15.843 DEBUG (MainThread) [google_nest_sdm.event] EventMessage raw_data={'eventId': 'd577b58d-21a6-40d5-b1ec-bac6f912639f', 'timestamp': '2023-10-18T11:00:08.347383Z', 'resourceUpdate': {'name': 'enterprises/[redacted]/devices/[redacted]--[redacted]', 'traits': {'sdm.devices.traits.ThermostatMode': {'mode': 'OFF', 'availableModes': ['OFF']}, 'sdm.devices.traits.ThermostatEco': {'availableModes': ['OFF', 'MANUAL_ECO'], 'mode': 'OFF', 'heatCelsius': 0.0, 'coolCelsius': 0.0}, 'sdm.devices.traits.ThermostatTemperatureSetpoint': {}}}, 'userId': '[redacted]', 'resourceGroup': ['enterprises/[redacted]/devices/[redacted]--[redacted]']}
2023-10-18 07:00:15.844 DEBUG (MainThread) [google_nest_sdm.device] Processing update d577b58d-21a6-40d5-b1ec-bac6f912639f @ 2023-10-18 11:00:08.347383+00:00
2023-10-18 07:00:15.845 DEBUG (MainThread) [google_nest_sdm.device] Trait update {'sdm.devices.traits.ThermostatMode': {'mode': 'OFF', 'availableModes': ['OFF']}, 'sdm.devices.traits.ThermostatEco': {'availableModes': ['OFF', 'MANUAL_ECO'], 'mode': 'OFF', 'heatCelsius': 0.0, 'coolCelsius': 0.0}, 'sdm.devices.traits.ThermostatTemperatureSetpoint': {}, 'name': 'enterprises/[redacted]/devices/[redacted]--[redacted]'}
2023-10-18 07:00:15.855 INFO (MainThread) [homeassistant.components.automation.debug_thermostat_off] Debug - thermostat off: Running automation actions
2023-10-18 07:00:15.856 INFO (MainThread) [homeassistant.components.automation.debug_thermostat_off] Debug - thermostat off: Executing step call service
2023-10-18 07:00:15.857 INFO (MainThread) [homeassistant.components.automation.debug_thermostat_off] Debug - thermostat off: Executing step call service
2023-10-18 07:00:15.860 DEBUG (MainThread) [homeassistant.components.nest] Stopping nest subscriber
2023-10-18 07:00:15.874 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'homeassistant.util.read_only_dict.ReadOnlyDict object' has no attribute 'current_temperature' when rendering '{{ state.attributes.current_temperature }}'
2023-10-18 07:00:15.874 WARNING (MainThread) [homeassistant.components.homeassistant.triggers.numeric_state] Error in 'Fish Tank Heater ON' trigger: In 'numeric_state' condition: entity climate.living_room_thermostat state '' cannot be processed as a number
2023-10-18 07:00:15.888 INFO (Thread-ConsumeBidirectionalStream) [google.api_core.bidi] Thread-ConsumeBidirectionalStream exiting
2023-10-18 07:00:16.609 DEBUG (ThreadPoolExecutor-1_0) [google_nest_sdm.google_nest_subscriber] Subscriber 'projects/tough-medley-[redacted]/subscriptions/home-assistant-[redacted]' configured on topic 'projects/sdm-prod/topics/enterprise-[redacted]'
2023-10-18 07:00:16.623 DEBUG (MainThread) [google_nest_sdm.google_nest_subscriber] Starting background watchdog thread
2023-10-18 07:00:16.624 DEBUG (MainThread) [google_nest_sdm.auth] request[get]=https://smartdevicemanagement.googleapis.com/v1/enterprises/[redacted]/structures
2023-10-18 07:00:16.997 DEBUG (MainThread) [google_nest_sdm.auth] response={'structures': [{'name': 'enterprises/[redacted]/structures/[redacted]-[redacted]', 'traits': {'sdm.structures.traits.Info': {'customName': '20 Holt Ct'}}}]}
2023-10-18 07:00:16.998 DEBUG (MainThread) [google_nest_sdm.auth] request[get]=https://smartdevicemanagement.googleapis.com/v1/enterprises/[redacted]/devices
2023-10-18 07:00:17.726 DEBUG (MainThread) [google_nest_sdm.auth] response={'devices': [{'name': 'enterprises/[redacted]/devices/[redacted]', 'type': 'sdm.devices.types.THERMOSTAT', 'assignee': 'enterprises/[redacted]/structures/[redacted]-[redacted]/rooms/[redacted]', 'traits': {'sdm.devices.traits.Info': {'customName': 'Living Room Thermostat'}, 'sdm.devices.traits.Humidity': {'ambientHumidityPercent': 60}, 'sdm.devices.traits.Connectivity': {'status': 'ONLINE'}, 'sdm.devices.traits.Fan': {'timerMode': 'OFF'}, 'sdm.devices.traits.ThermostatMode': {'mode': 'HEAT', 'availableModes': ['HEAT', 'COOL', 'HEATCOOL', 'OFF']}, 'sdm.devices.traits.ThermostatEco': {'availableModes': ['OFF', 'MANUAL_ECO'], 'mode': 'OFF', 'heatCelsius': 16.666672, 'coolCelsius': 24.592285}, 'sdm.devices.traits.ThermostatHvac': {'status': 'HEATING'}, 'sdm.devices.traits.Settings': {'temperatureScale': 'FAHRENHEIT'}, 'sdm.devices.traits.ThermostatTemperatureSetpoint': {'heatCelsius': 20.442734},
You’ll notice the thermostat available modes were messed up (only showing OFF) which triggered an automation to reboot the integration and in this example it solved the problem and the right values now came through and integration started working fine again.
Hopefully Google can solve soon.
allenporter
commented
11 months ago @dredjohn21 right but my point is the integration already won't change the HVAC modes when that happens, after a recent fix. What version of HA? what is the automation detecting?
dredjohn21
commented
10 months ago I'm now using:
Home Assistant 2023.10.3 Supervisor 2023.10.0 Operating System 11.0 Frontend 20231005.0 - latest
Looking at the recent activity it appears to be happening when I have automations set to change the temperature (e.g. when we wake up in morning, turn on the heat)
service: climate.set_temperature
data:
hvac_mode: heat
temperature: 70
target:
device_id: [redacted]It doesn't always happen when the automation changes temperatures... but it sometimes does... so maybe at the time it's setting temp it checks on available HVAC modes and when it sees "heat" isn't available it turns OFF? (that's the Google bug they hopefully will fix)
My automation to alert me of problems was coded based on the thermostat turning OFF, which the logs above show happens when the available HVAC options are not coming through properly.
aredon
commented
9 months ago Unsure if this is related but my nest integration has started getting stuck at initializing after a HASS/router restart. I've also restarted the nest itself a couple times to no avail. This message is flooding the log something fierce (we're talking >700 entries a minute).
Logger: google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager
Source: /usr/local/lib/python3.11/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py:131
First occurred: 7:47:26 PM (9 occurrences)
Last logged: 7:47:28 PM
Top-level exception occurred in callback while processing a message
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 126, in _wrap_callback_errors
callback(message)
File "/usr/local/lib/python3.11/site-packages/google_nest_sdm/google_nest_subscriber.py", line 283, in callback_wrapper
future.result()
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.11/site-packages/google_nest_sdm/google_nest_subscriber.py", line 533, in _async_message_callback
event = EventMessage(payload, self._auth)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/google_nest_sdm/event.py", line 346, in __init__
super().__init__(**raw_data, auth=auth)
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for EventMessage
relationUpdate -> subject
field required (type=value_error.missing)
@aredon please file a new issue and include all relevant information in the issue template.
cengel74
commented
8 months ago There hasn't been much activity in here but I wanted to pop in and say I'm still having the same issue and have been for quite some time. I posted data from this morning into the google issue : https://issuetracker.google.com/issues/291659614?pli=1
domoritz
commented
8 months ago I got emailed by Google people to ask for more logs and I sent them details months ago. Nothing has been fixed yet as far as I can see.
issue-triage-workflows[bot]
commented
5 months ago There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
domoritz
commented
5 months ago This is still a problem. I can turn the fan mode on but not off.
dredjohn21
commented
5 months ago This is stil broken - google asked for logs months ago and never did anything. Very frustrating. Please don’t close.
ianwood
commented
4 months ago API is still bombing out at least a few times a day. I've written an exception handler in Node Red to at the beginning of my HVAC routines to verify the thermostat is actually in the right mode before trying to set it. If it is reported as off when it isn't supposed to be, it reloads the integration which fixes the issue without human intervention. But there are still automations that get caught out from time to time. Minor inconvenience but would be better if Google just actually fixed it.
Nxt3
commented
4 months ago API is still bombing out at least a few times a day. I've written an exception handler in Node Red to at the beginning of my HVAC routines to verify the thermostat is actually in the right mode before trying to set it. If it is reported as off when it isn't supposed to be, it reloads the integration which fixes the issue without human intervention. But there are still automations that get caught out from time to time. Minor inconvenience but would be better if Google just actually fixed it.
Could you please share how you are reloading only the Nest integration via node red?
cengel74
commented
4 months ago I gave up on waiting and bought an Ecobee which has been rock solid with local control. Sold my nest on ebay.
pnbruckner
commented
4 months ago I meant to do a lot more digging, but I just haven't had the time. But I found the following, which might explain why messages are being received out of order. I still think that needs to be resolved. Any attempt to throw away out of order messages is only making the integration not really understand the state of the Nest devices. At least, that's my "2 cents."
So, here is where I believe the subscription is being created:
Here is the code for convenience:
message_retention_duration = Duration()
message_retention_duration.FromSeconds(DEFAULT_MESSAGE_RETENTION_SECONDS)
subscription_request = {
"name": subscription_name,
"topic": topic_name,
"message_retention_duration": message_retention_duration,
}
_LOGGER.debug(f"Creating subscription: {subscription_request}")
subscriber.create_subscription(request=subscription_request)And here is what I found for creating a subscription with ordering:
Again, for convenience:
def create_subscription_with_ordering(
project_id: str, topic_id: str, subscription_id: str
) -> None:
"""Create a subscription with ordering enabled."""
# [START pubsub_enable_subscription_ordering]
from google.cloud import pubsub_v1
# TODO(developer): Choose an existing topic.
# project_id = "your-project-id"
# topic_id = "your-topic-id"
# subscription_id = "your-subscription-id"
publisher = pubsub_v1.PublisherClient()
subscriber = pubsub_v1.SubscriberClient()
topic_path = publisher.topic_path(project_id, topic_id)
subscription_path = subscriber.subscription_path(project_id, subscription_id)
with subscriber:
subscription = subscriber.create_subscription(
request={
"name": subscription_path,
"topic": topic_path,
"enable_message_ordering": True,
}
)
print(f"Created subscription with ordering: {subscription}")
# [END pubsub_enable_subscription_ordering]Notice how this snipped includes "enable_message_ordering": True in the subscription request, whereas the code in google_nest_sdm.google_nest_subscriber does not.
allenporter
commented
4 months ago I am familiar with that. As I said before, lets get more specific though about an ordering problem you are seeing and we can discuss solutions for it. Please record the set of messages you receive, showing the ordering problem that it is causing.
I didn't want to get into the weeds of the implementation details here before which is why I suggested focusing on the specific problem you are seeing. I said before we're not doing anything special about delivery order, but that isn't exactly true. To be clear, it is normal and expected for messages to be received out of order. We are following the API guide here https://developers.google.com/nest/device-access/api/events:
Pub/Sub does not guarantee ordered delivery of events, and the receipt order of events may not correspond to the order in which the events actually occurred. Use the timestamp field to aid in reconciliation of event order. Events may also arrive individually or combined into a single event message.and it links to documentation about message ordering. So we follow the advice there https://github.com/allenporter/python-google-nest-sdm/blob/9646bde0b20c927f1f2e2009ce86ff4c2c12f3c4/google_nest_sdm/device.py#L296 and use timestamp to aid in reconciliation of event order.
Regarding the ordering feature itself in pubsub my impression is the ordering documentation states To receive a set of ordered messages in a region, you must publish all messages with the same ordering key in the same region -- that is, this is a feature of the publisher. Nest SDM API does not suggest using ordered messages and as the ordering documentation states to use the timestamp instead.
Again, if you want to talk about message ordering then lets talk about specifics of out of order messages you are receiving that are causing problems as requested above.
allenporter
commented
3 months ago https://issuetracker.google.com/issues/291659614 has an update that a fix is rolling out to prodution for the issue where thermostates appear off, and they request you provide update there if you see this issue.
allenporter
commented
2 months ago Google team says "The team has successfully implemented a fix, which is currently being rolled out to the production environment. However, due to the complexity of the issue, it's possible that this fix may resolve the problem for some thermostat models while others may still experience it. If you're still encountering issues with the Thermostat mode randomly reporting "Off" status, we kindly ask for your assistance in troubleshooting by providing us with most recent logs (including timestamps and time zone information) along with the model information of your thermostats."
I am closing since and you can eport status on https://issuetracker.google.com/issues/291659614 if you are seeing an issue.
Thank you to everyone who took the time to report information to Google. This issue never affected me personally so it really needed your patience and support to resolve.
The problem
Nest thermostat integration (using the Google Cloud API) works normally for several hours or even days but then fails and the integration needs to be reloaded to work again. It seems that the more recent versions of core have made this problem worse as I am now having to restart the integration almost daily.
Integration will show as "Off" with no other modes when it has failed. Reloading the integration restores the normal HVAC modes. Looking at the history for the device when the integration has failed, it will show that it was turned off at some point earlier although there was nothing commanding it to do so.
What version of Home Assistant Core has the issue?
core-2023.4.5
What was the last working version of Home Assistant Core?
No response
What type of installation are you running?
Home Assistant OS
Integration causing the issue
Nest
Link to integration documentation on our website
https://www.home-assistant.io/integrations/nest/
Diagnostics information
{ "home_assistant": { "installation_type": "Home Assistant OS", "version": "2023.4.5", "dev": false, "hassio": true, "virtualenv": false, "python_version": "3.10.10", "docker": true, "arch": "x86_64", "timezone": "America/Los_Angeles", "os_name": "Linux", "os_version": "6.1.24", "supervisor": "2023.04.1", "host_os": "Home Assistant OS 10.0", "docker_version": "23.0.3", "chassis": "embedded", "run_as_root": true }, "custom_components": { "frigate": { "version": "4.0.0", "requirements": [ "pytz==2022.7" ] }, "hacs": { "version": "1.32.1", "requirements": [ "aiogithubapi>=22.10.1" ] } }, "integration_manifest": { "domain": "nest", "name": "Google Nest", "after_dependencies": [ "media_source" ], "codeowners": [ "@allenporter" ], "config_flow": true, "dependencies": [ "ffmpeg", "http", "application_credentials" ], "dhcp": [ { "macaddress": "18B430" }, { "macaddress": "641666" }, { "macaddress": "D8EB46*" } ], "documentation": "https://www.home-assistant.io/integrations/nest", "iot_class": "cloud_push", "loggers": [ "google_nest_sdm", "nest" ], "quality_scale": "platinum", "requirements": [ "python-nest==4.2.0", "google-nest-sdm==2.2.4" ], "is_built_in": true }, "data": { "data": { "name": "REDACTED", "type": "sdm.devices.types.THERMOSTAT", "assignee": "REDACTED", "traits": { "sdm.devices.traits.Info": { "customName": "REDACTED" }, "sdm.devices.traits.Humidity": { "ambientHumidityPercent": 64 }, "sdm.devices.traits.Connectivity": { "status": "ONLINE" }, "sdm.devices.traits.Fan": { "timerMode": "OFF" }, "sdm.devices.traits.ThermostatMode": { "mode": "HEATCOOL", "availableModes": [ "HEAT", "COOL", "HEATCOOL", "OFF" ] }, "sdm.devices.traits.ThermostatEco": { "availableModes": [ "OFF", "MANUAL_ECO" ], "mode": "OFF", "heatCelsius": 15.555527, "coolCelsius": 29.444397 }, "sdm.devices.traits.ThermostatHvac": { "status": "OFF" }, "sdm.devices.traits.Settings": { "temperatureScale": "FAHRENHEIT" }, "sdm.devices.traits.ThermostatTemperatureSetpoint": { "heatCelsius": 20.555555, "coolCelsius": 23.333334 }, "sdm.devices.traits.Temperature": { "ambientTemperatureCelsius": 20.839996 } }, "parentRelations": [ { "parent": "REDACTED", "displayName": "REDACTED" } ] }, "command": { "sdm.devices.commands.ThermostatMode.SetMode_count": 1, "sdm.devices.commands.ThermostatMode.SetMode_sum": 755, "sdm.devices.commands.Fan.SetTimer_count": 1, "sdm.devices.commands.Fan.SetTimer_sum": 771 }, "event_media": { "event": 6 } } }
Example YAML snippet
No response
Anything in the logs that might be useful for us?
Additional information
No response