Withings - Showing as Unknown and/or Showing Detected even when not in bed

oneseventhree commented 8 months ago

The problem

Sensor shows as unavailable intermittently:

The old Withings integration I never had issues with (apart from installation) but after it was installed it worked perfetly for years. This new integration is a bit more buggy.

I've turned on loggings so hopefully something shows up

What version of Home Assistant Core has the issue?

core-2023.10.3

What was the last working version of Home Assistant Core?

Pre core-2023.10.x

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Withings

Link to integration documentation on our website

https://www.home-assistant.io/integrations/withings/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 8 months ago

Hey there @joostlek, mind taking a look at this issue as it has been labeled with an integration (withings) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `withings` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign withings` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

withings documentation withings source _{^{(message by IssueLinks)}}

joostlek commented 8 months ago

Do you use home assistant cloud or is your home assistant exposed to the public?

oneseventhree commented 8 months ago

Do you use home assistant cloud or is your home assistant exposed to the public?

I have nabu casa :)

joostlek commented 8 months ago

In that case it should just work. Please turn on debug logging and refresh the integration.

feacohen commented 8 months ago

I have the same Problem. I use duckdns and the bed sensore is always unknown.

lukas-clarke commented 8 months ago

I'm also seeing this issue after upgrading from 2023.9.x to 2023.10.5. I used a reverse proxy for remote access, and I'm seeing the bed sensor intermittently go unknown when someone is not in the bed.

oneseventhree commented 8 months ago

In that case it should just work. Please turn on debug logging and refresh the integration.

how do I get the debug log?

There has been a few times while in bed the integration goes to "unknown". Also when I get out of bed it still shows me in bed. These are intermittenent and were not present during the old integration

joostlek commented 8 months ago

You can get the debug log by turning off debug logging at integrations -> Withings -> enable debug logging and then reload your config entry

oneseventhree commented 8 months ago

You can get the debug log by turning off debug logging at integrations -> Withings -> enable debug logging and then reload your config entry

I woke up this morning - got out of bed and the sensor said I was still in bed. I turned on debug logging on and reloaded integration and its now showing unknown. Log attached

home-assistant_2023-10-24T19-57-23.394Z.log

joostlek commented 8 months ago

Yes that means it works, since it received the in bed message, but it didn't receive the out bed one. After reloading the unavailable is expected

oneseventhree commented 8 months ago

Yes that means it works, since it received the in bed message, but it didn't receive the out bed one. After reloading the unavailable is expected

How can I fix it so that it shows out of bed? This has only been an issue in the new integration

joostlek commented 8 months ago

I mean, I find it strange that it only works for in bed and not for out bed. It looks like you got an error code from Withings, 380, but I am not sure what that means. The webhook itself seems to be made correctly using Home Assistant Cloud. Maybe try a full restart or wait till 2023.11, dropping next week. In that release the integration is even more improved and logs will be more useful. (Or test the beta later today if you feel like it).

(PS: If you don't need to debug alexa anymore, I would suggest turning the debug logging of for them as the integration is shitting in your logs making them bigger than needed imo)

oneseventhree commented 8 months ago

I mean, I find it strange that it only works for in bed and not for out bed. It looks like you got an error code from Withings, 380, but I am not sure what that means. The webhook itself seems to be made correctly using Home Assistant Cloud. Maybe try a full restart or wait till 2023.11, dropping next week. In that release the integration is even more improved and logs will be more useful. (Or test the beta later today if you feel like it).

(PS: If you don't need to debug alexa anymore, I would suggest turning the debug logging of for them as the integration is shitting in your logs making them bigger than needed imo)

I have tried a full power cycle and still it does it intermittenly.

Thanks for the logging tip - I'll remove alexa.

Also, most of the time I restart home assistant I get this and have to reauth Withings:

joostlek commented 8 months ago

Do you know if you actually get new sleep data in the sensors? Do you have 2 instances of HA running? Or using your Withings login somewhere else?

oneseventhree commented 8 months ago

Do you know if you actually get new sleep data in the sensors? Do you have 2 instances of HA running? Or using your Withings login somewhere else?

I only have the bed sensor enabled - What should I enable to check sleep data?

I only have 1 instance of HA - Withings login only on my phone but I never open the app

joostlek commented 8 months ago

Any sleep entity. Those are disabled by default.

I find this issue strange and since a lot has changed I think it's best to test with the beta release (released today) or the 2023.11, released next week.

oneseventhree commented 8 months ago

Any sleep entity. Those are disabled by default.

I find this issue strange and since a lot has changed I think it's best to test with the beta release (released today) or the 2023.11, released next week.

I have updated and the first time getting out of bed with the new update still shows me in bed even though I am out. Lucky I had debugging on so I reloaded the integration and attached thte debug:

home-assistant_withings_2023-11-02T20-53-35.984Z.log

Looking forward to hearing what you find :)

joostlek commented 8 months ago

Wait, do you have some automation for enabling and disabling cloud access?

oneseventhree commented 8 months ago

Wait, do you have some automation for enabling and disabling cloud access?

Nope - I wouldn’t even know how to do that haha

joostlek commented 8 months ago

Like, for some reason, the integration gets a signal that the cloud is connected and thus starts setting up cloudhooks. In that process it unregisters the old webhooks (which are also cloudhooks) and then withings returns an error because apparently the webhook is not found at their side which is strange.

So there are 2 things I can't place: why does it want to setup cloud all of a sudden? And why does withings fail while it shouldn't.

oneseventhree commented 8 months ago

Like, for some reason, the integration gets a signal that the cloud is connected and thus starts setting up cloudhooks. In that process it unregisters the old webhooks (which are also cloudhooks) and then withings returns an error because apparently the webhook is not found at their side which is strange.

So there are 2 things I can't place: why does it want to setup cloud all of a sudden? And why does withings fail while it shouldn't.

Is there anything I can do to help solve it? I never had these issues with the old integration

joostlek commented 8 months ago

Did I already ask for debug logging from reload until the point this happens?

oneseventhree commented 8 months ago

:> Did I already ask for debug logging from reload until the point this happens?

Yep - that was the log above in: https://github.com/home-assistant/core/issues/102359#issuecomment-1791524562 Where the integration was stuck on "in" bed.

This morning the integration worked perfectly. It even showed me in bed all night instead of in and out. The log is here: home-assistant_withings_2023-11-04T22-46-42.700Z.log

I turned off debugging to get that above log.

So I turned debugging back on reloaded the integration and turned debugging off: [Uploading home-assistant_withings_2023-11-04T22-46-56.145Z.log…]()

Both these new logs show what its meant to do when its working correctly.

I am not sure why its so intermittent. It's like every second day the bed sensor gets stuck on "in"

oneseventhree commented 8 months ago

Did I already ask for debug logging from reload until the point this happens?

Woke up again this morning - got out of bed and the sensor still showed as I was in. Reloadeded the automation at 7:37am because I already had debugging on - sensor went unknown - went back to bed and it showed as detected then got out of bed and it showed it as clear: home-assistant_withings_2023-11-05T20-44-43.165Z.log

joostlek commented 8 months ago

I am still really curious how this happens. Currently I am a bit busy as I have a big deadline coming and I am going to Poland for a conference after that, so I am a bit short on time. I will try to ask around to some people who work with cloudhooks and check if this sounds familiar.

oneseventhree commented 7 months ago

I am still really curious how this happens. Currently I am a bit busy as I have a big deadline coming and I am going to Poland for a conference after that, so I am a bit short on time. I will try to ask around to some people who work with cloudhooks and check if this sounds familiar.

Thank you :) I’ve been doing some testing myself. If I don’t restart HA the first day the sensor messes up but then everyday after that it’s fine hence why it seemed intermittent. I was doing a restart a day because of adding HACS addons. When I stopped for a few days the sensor worked mostly all the time

naschine commented 7 months ago

Hi everyone! I had the same problem and decided to delete my entries for the Withings integration.

I have two bed sensors on separate accounts, which did work with the usual hackery. After I deleted them and recreated one, checking the cloudhook and updating the callback URL on the dev portal, it seems to work.

But the "in_bed" binary sensor is gone!? I tried to delete and recreate it again, but it just won't show up.

Latest log entries after I recreated it look rather good from what I can tell. Subscribing to the hook does work as well it seems.

2023-11-14 15:26:38.371 DEBUG (MainThread) [homeassistant.components.withings] Finished fetching Withings measurements data in 0.097 seconds (success: True)
2023-11-14 15:26:38.472 DEBUG (MainThread) [homeassistant.components.withings] Finished fetching Withings workout data in 0.100 seconds (success: True)
2023-11-14 15:26:38.553 DEBUG (MainThread) [homeassistant.components.withings] Finished fetching Withings sleep data in 0.081 seconds (success: True)
2023-11-14 15:26:38.717 DEBUG (MainThread) [homeassistant.components.withings] Finished fetching Withings activity data in 0.163 seconds (success: True)
2023-11-14 15:26:38.718 DEBUG (MainThread) [homeassistant.components.withings] Finished fetching Withings bed presence data in 0.000 seconds (success: True)

Do I have to do anything else to get the in_bed binary sensor back? I already did activate all entities of the integration (25 that is).

Thanks in advance!

joostlek commented 7 months ago

The in bed sensor is hidden by default unless we receive a webhook call from Withings. I did this for 2 reasons, 1. If people dont have the bed sensors, they dont have to see them. 2. If your webhook isn't working for some reason you can recognize that

naschine commented 7 months ago

Oh, ok. So I should lay down on the sensor and see if it shows up?

joostlek commented 7 months ago

Yep

selfish commented 7 months ago

Hey everyone

I just wanted to drop in and share something about this issue. I've been watching it for a while because I ran into similar problems with the webhooks repeatedly stopping after some time.

What worked for me was turning off the "my home assistant" stuff and setting up the webhooks with my Nabu Casa URL. I did this about a week ago, and everything has been running smoothly since then (even after restarts, etc.). I don't know why this makes a difference, but it might help.

Cheers

oneseventhree commented 7 months ago

Hey everyone

I just wanted to drop in and share something about this issue. I've been watching it for a while because I ran into similar problems with the webhooks repeatedly stopping after some time.

What worked for me was turning off the "my home assistant" stuff and setting up the webhooks with my Nabu Casa URL. I did this about a week ago, and everything has been running smoothly since then (even after restarts, etc.). I don't know why this makes a difference, but it might help.

Cheers

The problems from my first post was having my Nabu address as my Home Assistant URL:

A few minutes ago I have turned on "Use Home Assistant Cloud":

and will monitor if it works better or not.

Are you saying you have done the opposite @selfish?

selfish commented 7 months ago

@oneseventhree My configuration is all in YAML, so I'm not entirely sure where the screens you're showing are. If you tell me where to find that, I'm happy to check my config.

With that said, I have my YAML setup like this:

homeassistant:
  internal_url: https://<-redacted->.ui.nabu.casa
  external_url: https://<-redacted->.ui.nabu.casa

And I removed default settings from my instance and instead put everything in manually like so:

# default_config:
assist_pipeline:
bluetooth:
config:
conversation:
counter:
dhcp:
energy:
hardware:
history:
homeassistant_alerts:
cloud:
image_upload:
input_boolean:
input_button:
input_datetime:
input_text:
logbook:
logger:
map:
media_source:
mobile_app:
#my:
network:
person:
schedule:
ssdp:
stream:
sun:
system_health:
tag:
timer:
usb:
webhook:
zeroconf:

python_script:
wake_on_lan:

# the rest doesn't change

(restart after doing that, of course)

Then, for the Withings app settings, I used https://<-redacted->.ui.nabu.casa/auth/external/callback as my URL.

After doing that, I removed all previous Withings integration instances, and I logged in to my HA instance using the https://<-redacted->.ui.nabu.casa URL and set them up again from there.

joostlek commented 7 months ago

I mean, it would be a valid workaround, I wouldn't support this method for the long run, but if this can help me debug the issue more that would be great

naschine commented 7 months ago

naschine: Oh, ok. So I should lay down on the sensor and see if it shows up? joostlek: Yep

Follow up to my previous comment:

The binary sensor did show up as expected. Also it did work for the first couple events.

But now I'm at work and out of bed since 6:30am and it still shows as I'm in bed.

I also use the external nabu casa URL instead of the "Use Home Assistant Cloud"-switch. That was part of the old Withings integration magic, to make it work.

Is there anything I can provide to give better insight?

Thanks for your work on the integration @joostlek

oneseventhree commented 7 months ago

naschine: Oh, ok. So I should lay down on the sensor and see if it shows up? joostlek: Yep

Follow up to my previous comment:

The binary sensor did show up as expected. Also it did work for the first couple events.

But now I'm at work and out of bed since 6:30am and it still shows as I'm in bed.

I also use the external nabu casa URL instead of the "Use Home Assistant Cloud"-switch. That was part of the old Withings integration magic, to make it work.

Is there anything I can provide to give better insight?

Thanks for your work on the integration @joostlek

Im glad it’s not just me!

I’ve switched to home assistant cloud URL. I’ll see how it goes tomorrow (9:30pm here now!

naschine commented 7 months ago

Update: Just got a mail from Withings, that says my server is not reachable with the external nabu casa url, with a timeout of 2 seconds.

That's strange... I'm currently on my HA setup from work via nabu casa.

Edit: I meant nabu casa hook url

naschine commented 7 months ago

One night passed again and the sensor received the event that I'm in bed, but not the event that I'm out of it.

Also I was spammed with mails from Withings yesterday, that my server isn't reachable. Even though I could test the URL in the Withings developer portal no problem and everything's greenlit.

GWorswick commented 7 months ago

I've been following this thread and I have exactly the same problem, it's been fine for years before the recent update. I have 2 bed sensors and it seems to intermittently fail to register the bed sensors activation. It has stopped working at night and in the morning but when I manually test it in the morning it's fine usually by the time I go to bed at night it stops working. Yesterday after seeings @selfish post I had a look at my configuration.yaml. There was no my: section, I've never consciously removed this. I only really play with homeassistant when something breaks though so I may have missed it in an update at some point. I went to my dev console at Withings and changed my URI to the Nabucasa address hoping this would fix it and it seemed to work. When I went to bed last night both bed sensors activated the my night time routine "House asleep" seemed to work however when I woke up this morning neither of the bed sensors sensed either me or my partner getting up. I received 5 emails from Withings last week about a failed connection but nothing recently and this happens every day. I did reset my internet connection during the day yesterday between 8.43am and 9.24am as I asked my internet provider to check the connection and they found nothing wrong. I have attached the Log, i hope it helps shed some light. We have a routine that is called "Night Wee wee" when one of us gets up to go to the toilet and we have another routine when both of us get up called "House is awake". We were both in bed at 22:54 I'll reset the log today and try adding my: to the configuration.yaml to see if that makes a difference. The email from Withings said:

We cannot reach your server to a notification (timeout of 2 seconds). Below the details Userid : Url : https://hooks.nabu.casa/gAAAAABlT Please fix your url. We will delete this notification in 20 days. We will not remove the notification if the next calls are correct.

home-assistant_2023-11-16T07-05-20.040Z.log

GWorswick commented 7 months ago

short update: After HA not registering either of us getting up this morning I have added my: to my config.yaml. I have left the Nadbucasa address in the URI at Withings. I fixed some automations that were not connected but were making the log look messy, I rebooted HA and turned on debug logging for WIthings. I just tested both bed sensors and they register state change in HA properly. I'll leave it alone now for the rest of the day as its currently working and see what the debug log says tomorrow morning and update you.

GWorswick commented 7 months ago

So neither my partner or I touched the bed or HA settings yesterday. I have attached the log and it looks like every hour Withings sends "goals data" then around 6pm Withings webhooks are unregistered and Unsubscribed for some reason? We went to bed together at about 10.45pm and it didn't seem to run the house asleep command as I had to turn off the lights manually. So it looks to me like something that happened before we went to bed caused it to stop working? The "night wee wee" routine is started by a motion sensor in the room and disabled by getting back in bed so the errors about night wee wee already running seems to be because the bed sensors aren't turning off this automation. I got up at about 7am I hope the log helps home-assistant_2023-11-17T07-13-58.374Z.log

dcmeglio commented 7 months ago

Has there been any progress on this issue? It is continually happening to me as well. Worked flawlessly for years until these updates

joostlek commented 7 months ago

I'm still trying to cook something to debug or analyse the problem

mysterytoy2 commented 7 months ago

I have noticed that every time I restart home assistant the withings sleep sensor shows unavailable. Everything is up to date on my system. I can get on the bed and the sensor changes to occupied then clear when I get out. The sensor continues to work perfectly until I restart Home Assistant.

I tried to reload both sensors but that doesn't fix it. Only thing that fixes is when the state of the sensor changes and then it works normally.

A kludge would be in the automations if I could add Unavailable as one of the states then I could test for that before I test for Occupied or Clear. That would fix my problem as currently my automations fail when the sensor is unavailable. Thanks for our time.

oneseventhree commented 7 months ago

I have noticed that every time I restart home assistant the withings sleep sensor shows unavailable. Everything is up to date on my system. I can get on the bed and the sensor changes to occupied then clear when I get out. The sensor continues to work perfectly until I restart Home Assistant.

I tried to reload both sensors but that doesn't fix it. Only thing that fixes is when the state of the sensor changes and then it works normally.

A kludge would be in the automations if I could add Unavailable as one of the states then I could test for that before I test for Occupied or Clear. That would fix my problem as currently my automations fail when the sensor is unavailable. Thanks for our time

I noticed the same. Ever since I started this thread a few weeks ago I have been testing. All different ways.

My error normally came after I restarted HA and went to SLEEP. Not just into my bed but sleep for hours. I noticed if -I restarted HA -Reloaded the integration (because sometimes it wouldnt detect) -Jumped into bed for a few minutes to make the sensor turn to Detected -Jumped out to make it clear

It worked when I actually went to bed to sleep. It's like the error happens only when in bed for a long period of time after a restart without reloading and jumping in and out of bed to reset the sensors state.

I am yet to try the above out with these settings:

@selfish to answer your question before this can be found in Settings > System > Networks in the UI.

If I keep HA running for a few days without restarting the sensor seems to work it's only after the reset that it plays up

selfish commented 7 months ago

I have noticed that every time I restart Home Assistant the Withings sleep sensor shows unavailable.

@mysterytoy2 I solved this by creating this blueprint, which restores the state of multiple entities when they become unavailable or unknown. Note that it depends on this Python service

@oneseventhree, mine looks different. It's also disabled because it's set from the YAML config:

And lastly, @joostlek - thank you for your work. and for trying to solve the problems with the integrations. If needed, I would be very glad to assist in any way I can.

joostlek commented 7 months ago

Also, the fact that the sensor is unknown after a restart is how it works as when HA boots we don't know if you're in bed or not

selfish commented 7 months ago

About the sensor's 'unknown' state after a HA restart. I get that we can't be sure if someone's in bed when HA boots up, but I think it might be better if the sensor could remember its last state, kind of like how a temperature sensor does. It's not perfect, but at least it gives us something to work with until the next update, right?

To me, resetting to 'unknown' every time feels a bit off, almost like a bug. I even rigged up the automation mentioned above to work around this. What do you think about giving us the option to keep the last state after a restart? I feel like it could make things smoother and more user-friendly for some. I would really like to have this configurable.

joostlek commented 7 months ago

The downside is that the next update is the next moment somebody lays in bed. I'd rather have people take into account 3 possible states than only 2. (I think the automation trigger to say, if the user lays in bed won't even work since when the state used to be "in bed" and then HA gets the update that the user went to bed, the state didn't change thus doesn't trigger automations)

selfish commented 7 months ago

I understand your point of view, but this is slightly unconventional, and while this type of sensor is slightly unusual indeed, this doesn't take into account certain use cases.

In my personal use case, I often restart home assistant after my wife is asleep, then head to bed myself. Unless I use automation to restore the wife's in-bed state, my bedtime automation won't run when I enter the bed, because not all sensors indicate the status.

I rarely have any downtime, and more often than not, I do restarts, which last a brief moment, with a very low chance of state changes between them, so being able to configure this behavior to match my needs would be a very welcome addition, saving me the effort of maintaining supporting automation.

It can be a non-default option, but I see no harm in offering state persistence as a config option

home-assistant / core