search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
70k stars 29.07k forks source link

Withings - Showing as Unknown and/or Showing Detected even when not in bed #102359

Closed oneseventhree closed 7 months ago

oneseventhree commented 9 months ago

The problem

Sensor shows as unavailable intermittently:

image

The old Withings integration I never had issues with (apart from installation) but after it was installed it worked perfetly for years. This new integration is a bit more buggy.

I've turned on loggings so hopefully something shows up

What version of Home Assistant Core has the issue?

core-2023.10.3

What was the last working version of Home Assistant Core?

Pre core-2023.10.x

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Withings

Link to integration documentation on our website

https://www.home-assistant.io/integrations/withings/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

joostlek commented 8 months ago

I'd rather let the user decide how they handle this themselves. Maybe someone always knows that when the sensor is unknown after 8:00 that it should be out of bed and when someone is home and HA restarts after 0:00 it should be in bed. I'm not the person to decide that, so I'd rather have users figure out a way for themselves to decide.

Anyway, this issue isn't really a place to discuss this. Feel free to reach me on Discord

selfish commented 8 months ago

Thanks for considering my suggestion. I respect your decision, though I'm disappointed as I thought the feature could enhance user experiences in cases like mine. Nonetheless, I appreciate your commitment to the project and will keep other solutions for my setup. Thanks again for your efforts and time.

mysterytoy2 commented 8 months ago

Thanks, I’ll give that a try.

From: selfish Sent: Sunday, November 19, 2023 6:17 AM To: home-assistant/core Cc: mysterytoy2 ; Mention Subject: Re: [home-assistant/core] Withings - Showing as Unknown and/or Showing Detected even when not in bed (Issue #102359)

I have noticed that every time I restart Home Assistant the Withings sleep sensor shows unavailable.

@mysterytoy2 I solved this by creating this blueprint, which restores the state of multiple entities when they become unavailable or unknown. Note that it depends on this Python service

@oneseventhree, mine looks different. It's also disabled because it's set from the YAML config:

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

mysterytoy2 commented 8 months ago

Sure, so my suggestion would be to add unavailable as one of the possible states. That way I would test for that condition first otherwise any associated automation will consistently fail.

From: Joost Lekkerkerker Sent: Sunday, November 19, 2023 6:20 AM To: home-assistant/core Cc: mysterytoy2 ; Mention Subject: Re: [home-assistant/core] Withings - Showing as Unknown and/or Showing Detected even when not in bed (Issue #102359)

Also, the fact that the sensor is unknown after a restart is how it works as when HA boots we don't know if you're in bed or not

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

mysterytoy2 commented 8 months ago

Automations that are triggered by the Withings event will always work. I have other automation and actions that check to see if someone or everyone are in bed and they will always fail after the restart until the Withings state becomes available that is unless unavailable is added to the list of states so I can keep my other automations from failing. Thanks for your help. Great Integration. I really appreciate having it.

From: Joost Lekkerkerker Sent: Sunday, November 19, 2023 6:59 AM To: home-assistant/core Cc: mysterytoy2 ; Mention Subject: Re: [home-assistant/core] Withings - Showing as Unknown and/or Showing Detected even when not in bed (Issue #102359)

The downside is that the next update is the next moment somebody lays in bed. I'd rather have people take into account 3 possible states than only 2. (I think the automation trigger to say, if the user lays in bed won't even work since when the state used to be "in bed" and then HA gets the update that the user went to bed, the state didn't change thus doesn't trigger automations)

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

joostlek commented 8 months ago

Unavailable isn't the right state. I won't add that.

joostlek commented 8 months ago

But again, let's have this discussion on discord, not here

oneseventhree commented 8 months ago

Spoke to soon - got out of bed this morning but just checked and its showing I've been in bed for 22 hours. I started debugging - reloaded the integration: home-assistant_withings_2023-11-20T09-26-14.802Z.log

I've started debugging again in case it happens again

mysterytoy2 commented 7 months ago

Good Afternoon,

I am getting an error when I try to save the blueprint. Here is the error: Message malformed: extra keys not allowed @ data['target']

Thanks for your time.

Richard

From: selfish Sent: Sunday, November 19, 2023 6:17 AM To: home-assistant/core Cc: mysterytoy2 ; Mention Subject: Re: [home-assistant/core] Withings - Showing as Unknown and/or Showing Detected even when not in bed (Issue #102359)

I have noticed that every time I restart Home Assistant the Withings sleep sensor shows unavailable.

@mysterytoy2 I solved this by creating this blueprint, which restores the state of multiple entities when they become unavailable or unknown. Note that it depends on this Python service

@oneseventhree, mine looks different. It's also disabled because it's set from the YAML config:

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

naschine commented 7 months ago

Just a little update: Still the same for me with Core 11.3

From unknown to in bed works, but I'll never get up :D

dcmeglio commented 7 months ago

So correct me if I'm wrong - the error we're getting back from the API is you tried to delete something that didn't exist (getting object not found when trying to delete a webhook). Why don't you just trap it and ignore this? The goal of this code is to clean up an old webhook. If the API says it's already gone, why not handle this gracefully? I agree understanding the root cause would be good, but since in this case it's saying "yup it's already deleted" why are we failing everything vs. moving along?

joostlek commented 7 months ago

I mean that's probably going to be the thing, but I am still dazzled about the why. Can you maybe do something?

cd /config
curl -o- -L https://gist.githubusercontent.com/bdraco/43f8043cb04b9838383fd71353e99b18/raw/core_integration_pr | bash /dev/stdin -d withings -p 104579

This should install the Withings integration as custom integration, making you able to update stuff in this.

diff --git a/homeassistant/components/withings/__init__.py b/homeassistant/components/withings/__init__.py
index 701f7f444c..f9a20514ba 100644
--- a/homeassistant/components/withings/__init__.py
+++ b/homeassistant/components/withings/__init__.py
@@ -5,6 +5,7 @@ For more details about this platform, please refer to the documentation at
 from __future__ import annotations

 import asyncio
+import json
 from collections.abc import Awaitable, Callable
 import contextlib
 from dataclasses import dataclass, field
@@ -298,6 +299,8 @@ async def async_unsubscribe_webhooks(client: WithingsClient) -> None:
     """Unsubscribe to all Withings webhooks."""
     current_webhooks = await client.list_notification_configurations()

+    LOGGER.info(json.dumps(current_webhooks))
+
     for webhook_configuration in current_webhooks:
         LOGGER.debug(
             "Unsubscribing %s for %s in %s seconds",

If you modify this in custom_components/withings/__init__.py we can see what webhook its removing. Very curious what it returns.

dcmeglio commented 7 months ago

Sure, just installed it. And yeah I agree we should get to the underlying cause, but would also be good to get it working since it seems like it's a harmless error in this specific case

sjd17 commented 7 months ago

I'm having the same problem that also started in October. I'm using Nabu Casa. Usually it fails to register getting out of bed but sometimes in bed as well. When I notice my automation not running as expected, I reload the withings automation which seems to fix it for a few days before it starts having problems again. I haven't had time to do any real troubleshooting.

joostlek commented 7 months ago

@dcmeglio any updates?

mysterytoy2 commented 7 months ago

Hi Joost,

I haven’t tried restarting HA in about 10 days. I can try it again if you want me to. Thanks.

Richard

From: Joost Lekkerkerker Sent: Tuesday, December 5, 2023 8:20 AM To: home-assistant/core Cc: mysterytoy2 ; Mention Subject: Re: [home-assistant/core] Withings - Showing as Unknown and/or Showing Detected even when not in bed (Issue #102359)

@dcmeglio any updates?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

dcmeglio commented 7 months ago

Still monitoring. The issue hasn’t reoccurred yet

joostlek commented 7 months ago

Heh, I thought it happened ever so often.

Did anything else in your setup change?

oneseventhree commented 7 months ago

Heh, I thought it happened ever so often.

Did anything else in your setup change?

Mine has been working better the last few days and I haven’t made any changes. I can’t seem to fault it. I’ll keep monitoring.

GWorswick commented 7 months ago

Nothing has changed for me, it still failed to register me getting into bed last night. I thought @dcmeglio said it was a problem with an error about a webhook?. I'm not great with this sort of stuff but I would love a fix. This problem has already killed the idea of us getting curtain openers with my wife. She is getting more and more annoyed by it as its worked for years and now is so intermittent its destroying her faith in automating our home.

joostlek commented 7 months ago

Ouch, we don't want that. But feel free to also apply the debug method I posted above. If you can provide logs from the next time it happens, that'd be great.

GWorswick commented 7 months ago

@joostlek Apologies I'm not great with all this technical stuff. Do I run the curl command from the terminal in HA? Then edit the file custom_components/withings/init.py using file editor so that it has the same stuff in it that is in your post above? I assume after I've done that you want me to turn on debugging?

Thanks :) I'm eager to get this working :)

joostlek commented 7 months ago

Exactly that!

naschine commented 7 months ago

Nothing's changed for me, still won't register any so often when I get out of bed.

Sleeping for days and the occasional restart :D

image

I'll try to add the integration as custom integration as well and see if my logs will show some interesting bits.

joostlek commented 7 months ago

Wait a second, this is specific issue is about the in bed sensor going to unknown. If you have different logs than @oneseventhree, please open a separate issue

naschine commented 7 months ago

The title says "and/or showing Detected even when not in bed", so why it would be a separate issue?

From my impression and what I've read most of the people here have the problem, that the detection of the correct state does not work anymore and not that the entity suddenly goes to "unknown".

And mostly so the event for getting out of bed and the entity stays on "detected".

But I can open a different issue as well if that helps, no problem.

joostlek commented 7 months ago

I'd rather have this on a separate issue. It's also not that I won't help you there, I want to keep the discussions separated. Since there is a specific error I am trying to chase here.

The debug method I proposed was also to try to debug that issue. So if you don't have the same problem as @oneseventhree in the logs (with the CLOUD_CONNECTED_STATE thingy) please create a separate one.

GWorswick commented 7 months ago

@joostlek Apologies again, I have run the curl command in terminal which worked fine. Do I just add the below to the bottom of the file custom_components/withings/init.py?

diff --git a/homeassistant/components/withings/init.py b/homeassistant/components/withings/init.py index 701f7f444c..f9a20514ba 100644 --- a/homeassistant/components/withings/init.py +++ b/homeassistant/components/withings/init.py @@ -5,6 +5,7 @@ For more details about this platform, please refer to the documentation at from future import annotations

import asyncio +import json from collections.abc import Awaitable, Callable import contextlib from dataclasses import dataclass, field @@ -298,6 +299,8 @@ async def async_unsubscribe_webhooks(client: WithingsClient) -> None: """Unsubscribe to all Withings webhooks.""" current_webhooks = await client.list_notification_configurations()

joostlek commented 7 months ago

You should add the green lines to the code

joostlek commented 7 months ago

If you need more help, please reach out on discord

joostlek commented 7 months ago

@oneseventhree what's your timezone?

In the log @GWorswick provided to me on discord, the webhooks reload around 18:02 GMT. I remember yours reload around 7:00, but which timezone?

Maybe there's a coincidence there.

oneseventhree commented 7 months ago

@oneseventhree what's your timezone?

In the log @GWorswick provided to me on discord, the webhooks reload around 18:02 GMT. I remember yours reload around 7:00, but which timezone?

Maybe there's a coincidence there.

Australian Eastern Daylight Time (GMT+11)

I just checked my logs and for a week it looks like my sensor has been working. I’ll keep monitoring

joostlek commented 7 months ago

I need to recheck your logs (am in the bus so I can't check since the GitHub app is bugged) but yours triggered 7:00.

So this leads me to 2 things:

  1. Why do they reset at 18:00 GMT?
  2. Why don't I have this issue? I use HA cloud as well, but without bed sensor, but that should not really matter with why this reset is triggered. (If it would, I would expect that it would be more related to the bed sensor, eg reset after 1 hour after laying in bed or something).

Can you maybe remember anything you changed over the last days? Any custom integration that got updated/removed?

GWorswick commented 7 months ago

I have 2 bed sensors, i don't know if thats a reason?

oneseventhree commented 7 months ago

I need to recheck your logs (am in the bus so I can't check since the GitHub app is bugged) but yours triggered 7:00.

So this leads me to 2 things:

  1. Why do they reset at 18:00 GMT?
  2. Why don't I have this issue? I use HA cloud as well, but without bed sensor, but that should not really matter with why this reset is triggered. (If it would, I would expect that it would be more related to the bed sensor, eg reset after 1 hour after laying in bed or something).

Can you maybe remember anything you changed over the last days? Any custom integration that got updated/removed?

I updated some custom HAC integrations as per normal a the OS update. Do you want me to start the debug log again?

joostlek commented 7 months ago

Can you list your custom integrations?

dcmeglio commented 7 months ago

Only change on my end is I’m using the 23.12 beta. But even if that had fixes in the withings integration wouldn’t have them since it’s now a custom integration. So just coincidence I guess?

joostlek commented 7 months ago

I don't think I have changed anything major in 2023.12.

But I am a bit puzzled about how not everyone has this issue.

oneseventhree commented 7 months ago

Can you list your custom integrations?

Dammit. I jinxed myself. Got out of bed this morning and its still showing me in there after 1 hour. Only thing I did was enable debugging before bed last night. No restart - no updating of integrations: home-assistant_withings_2023-12-06T20-42-09.196Z.log

I then reloaded the integration and captured the debug incase you need that too: home-assistant_withings_2023-12-06T20-42-28.455Z.log

Going to update to 2023.12.0 now

joostlek commented 7 months ago

I'm currently on mobile so I can't search in your logs, but did apply those debug logs?

GWorswick commented 7 months ago

Morning all :) @joostlek My integration worked last night and this morning, I have uploaded the log to your discord, ill leave it running again and see if it works again tonight.

joostlek commented 7 months ago

So my debug code had a bug in it which caused it to raise an error before this issue was happening. I made a new debug code:

diff --git a/homeassistant/components/withings/__init__.py b/homeassistant/components/withings/__init__.py
index 701f7f444c..5eb8a74c03 100644
--- a/homeassistant/components/withings/__init__.py
+++ b/homeassistant/components/withings/__init__.py
@@ -7,7 +7,7 @@ from __future__ import annotations
 import asyncio
 from collections.abc import Awaitable, Callable
 import contextlib
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, asdict
 from datetime import timedelta
 from typing import TYPE_CHECKING, Any

@@ -298,6 +298,9 @@ async def async_unsubscribe_webhooks(client: WithingsClient) -> None:
     """Unsubscribe to all Withings webhooks."""
     current_webhooks = await client.list_notification_configurations()

+    for webhook_configuration in current_webhooks:
+        LOGGER.info(asdict(webhook_configuration))
+
     for webhook_configuration in current_webhooks:
         LOGGER.debug(
             "Unsubscribing %s for %s in %s seconds",

Please revert the old code

joostlek commented 7 months ago

Okay so that bug I added in the debug code actually helped me get a better view of what's happening where. Apparently the cloud is disconnected for a split second (or 3) and since the webhook unregister process is 5 seconds long and the register webhook process is 35 (including the unregister process), they collide and do the same thing at the same time.

Turns out I could not reproduce this because apparently my internet is too stable.

joostlek commented 7 months ago

@GWorswick Is now collecting some more logs so I can figure out what the best way is to solve this, if anyone else wants to help as well:

cd /config
curl -o- -L https://gist.githubusercontent.com/bdraco/43f8043cb04b9838383fd71353e99b18/raw/core_integration_pr | bash /dev/stdin -d withings -p 105203
joostlek commented 7 months ago

I pushed a possible fix. If people suffering from this issue (and if you added my previous debug code, you still suffer from the issue) could run the above commands again. If I can get confirmation around tomorrow I can try to make it in 2023.12.1

dcmeglio commented 7 months ago

Sorry what do you need me to do again?

cd /config
curl -o- -L https://gist.githubusercontent.com/bdraco/43f8043cb04b9838383fd71353e99b18/raw/core_integration_pr | bash /dev/stdin -d withings -p 104579

?

joostlek commented 7 months ago

105203 at the end. Please run this code for a few hours with debug logging on and send the logs on here

GWorswick commented 7 months ago

@joostlek I'm running the new curl now, i reboot and put debugging back on

GWorswick commented 7 months ago

I rebooted and it hasn't registered my wife is still in bed :(

dcmeglio commented 7 months ago

Not fixed for me with the latest patch. Here are the debug logs (I redacted webhook urls and nabucasa urls, but did a find and replace to keep them consistent. Seems the problem is it's unsubscribing multiple times? FYI I do have 2 bed sensors. I saw at least one other person reporting he has 2 bed sensors too. Maybe that's part of the problem?

2023-12-09 22:07:41.404 INFO (MainThread) [hass_nabucasa.iot] Connection closed: Closed by server. None (None)TOKEN1_ID
2023-12-09 22:07:41.416 DEBUG (MainThread) [custom_components.withings] Unregister Withings webhook (WEBHOOK1_ID)
2023-12-09 22:07:41.421 DEBUG (MainThread) [custom_components.withings] Cloudconnection state changed to CloudConnectionState.CLOUD_DISCONNECTED
2023-12-09 22:07:41.421 DEBUG (MainThread) [custom_components.withings] Unregister Withings webhook (WEBHOOK2_ID)
2023-12-09 22:07:42.194 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN1_ID for 1 in 1.0 seconds
2023-12-09 22:07:42.196 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN2_ID for 1 in 1.0 seconds
2023-12-09 22:07:43.333 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN1_ID for 4 in 1.0 seconds
2023-12-09 22:07:43.343 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN2_ID for 4 in 1.0 seconds
2023-12-09 22:07:43.631 INFO (MainThread) [hass_nabucasa.iot] Connected
2023-12-09 22:07:43.642 DEBUG (MainThread) [custom_components.withings] Cloudconnection state changed to CloudConnectionState.CLOUD_CONNECTED
2023-12-09 22:07:43.643 DEBUG (MainThread) [custom_components.withings] Registered Withings webhook at hass: https://hooks.nabu.casa/TOKEN2_ID
2023-12-09 22:07:43.649 DEBUG (MainThread) [custom_components.withings] Cloudconnection state changed to CloudConnectionState.CLOUD_CONNECTED
2023-12-09 22:07:43.649 DEBUG (MainThread) [custom_components.withings] Registered Withings webhook at hass: https://hooks.nabu.casa/TOKEN1_ID
2023-12-09 22:07:43.867 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN1_ID for 4 in 1.0 seconds
2023-12-09 22:07:43.871 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN2_ID for 4 in 1.0 seconds
2023-12-09 22:07:44.476 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN1_ID for 16 in 1.0 seconds
2023-12-09 22:07:44.501 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN2_ID for 16 in 1.0 seconds
2023-12-09 22:07:45.110 ERROR (MainThread) [homeassistant.util.logging] Exception in manage_cloudhook when dispatching 'CLOUD_CONNECTION_STATE': (<CloudConnectionState.CLOUD_CONNECTED: 'cloud_connected'>,)
Traceback (most recent call last):
  File "/config/custom_components/withings/__init__.py", line 251, in manage_cloudhook
    await register_webhook(None)
  File "/config/custom_components/withings/__init__.py", line 239, in register_webhook
    await async_subscribe_webhooks(client, webhook_url)
  File "/config/custom_components/withings/__init__.py", line 282, in async_subscribe_webhooks
    await async_unsubscribe_webhooks(client)
  File "/config/custom_components/withings/__init__.py", line 320, in async_unsubscribe_webhooks
    await client.revoke_notification_configurations(
  File "/usr/local/lib/python3.11/site-packages/aiowithings/withings.py", line 383, in revoke_notification_configurations
    await self._request(
  File "/usr/local/lib/python3.11/site-packages/aiowithings/withings.py", line 133, in _request
    raise WithingsInvalidParamsError(error)
aiowithings.exceptions.WithingsInvalidParamsError: Object was not found

2023-12-09 22:07:45.147 ERROR (MainThread) [homeassistant.util.logging] Exception in manage_cloudhook when dispatching 'CLOUD_CONNECTION_STATE': (<CloudConnectionState.CLOUD_CONNECTED: 'cloud_connected'>,)
Traceback (most recent call last):
  File "/config/custom_components/withings/__init__.py", line 251, in manage_cloudhook
    await register_webhook(None)
  File "/config/custom_components/withings/__init__.py", line 239, in register_webhook
    await async_subscribe_webhooks(client, webhook_url)
  File "/config/custom_components/withings/__init__.py", line 282, in async_subscribe_webhooks
    await async_unsubscribe_webhooks(client)
  File "/config/custom_components/withings/__init__.py", line 320, in async_unsubscribe_webhooks
    await client.revoke_notification_configurations(
  File "/usr/local/lib/python3.11/site-packages/aiowithings/withings.py", line 383, in revoke_notification_configurations
    await self._request(
  File "/usr/local/lib/python3.11/site-packages/aiowithings/withings.py", line 133, in _request
    raise WithingsInvalidParamsError(error)
aiowithings.exceptions.WithingsInvalidParamsError: Object was not found

2023-12-09 22:07:45.628 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN1_ID for 44 in 1.0 seconds
2023-12-09 22:07:45.650 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN2_ID for 44 in 1.0 seconds
2023-12-09 22:07:46.783 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN1_ID for 50 in 1.0 seconds
2023-12-09 22:07:46.804 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN2_ID for 50 in 1.0 seconds
2023-12-09 22:07:47.932 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN1_ID for 51 in 1.0 seconds
2023-12-09 22:07:47.945 DEBUG (MainThread) [custom_components.withings] Unsubscribing https://hooks.nabu.casa/TOKEN2_ID for 51 in 1.0 seconds