Have a monitor.xxxx_None sensor with value undefined show up

[skynet01]

I noticed that I have monitor.xxxx_None (xxx being the name) with value "unknown" show up in a log which seems to be the cause of a lot of false positives. Maybe it creates incorrect confidence value?

This started happening since I added another node to my network to make things more stable. There are numerous times when person is marked offline but when I check AppDaemon I can see that at least one node is showing 100% confidence level.

Maybe this wouldn't be that big of an issue if scanning only happened when gateways are triggered instead of also do scans when nodes "appear" to go offline which does happen on wifi. Maybe have that at least as an option? So I have had automations trigger in the middle of the night. I think I am almost ready to switch back to manually wrangling my nodes.

Thanks for the help as usual.

[Odianosen25]

Hello @skynet01,

Apologies taken a while to respond, been so busy of late. On your issue, its kind of interesting. Is it possible the new one didn't have a location set initially?

On running the scan when it goes offline. It doesn't do it once offline as per say, it does run a timer based on the system_timeout setting which is 60 by default to clear the entities from that node. So you can increase this time, and see if it helps your issue.

Regards

EDIT: Also I have noticed this app will give a lot of potential issues to anyone using AD but not having the app pinned to a thread. if that is your case, do consider doing that.

[skynet01]

1) What do you mean by location? Do you mean the name? I created a clone of the node, then changed the name of it and brought it online. I rebooted all nodes as well as Home assistant. It uses the same credentials etc. I can check specific settings if you have specific ideas. 2) I'll update the system_timeout and see if that helps. I guess i can make system_timeout value really high to eliminate my second issue.

3) How do you pin AD to a thread? 4) Sent you some coffee monies ❤️

[Odianosen25]

@skynet01,

What do you mean by location? Do you mean the name? I created a clone of the node, then changed the name of it and brought it online. I rebooted all nodes as well as Home assistant. It uses the same credentials etc. I can check specific settings if you have specific ideas

Yes I meant name. If the name is None or by mistake was empty, that's the only reason I could think of that could have caused it to give you None

I'll update the system_timeout and see if that helps. I guess i can make system_timeout value really high to eliminate my second issue.

Yep that should work. If it doesn't, I can completely eliminate it and make it optional. But I am not sure it will be good idea, as it has to clear the sensors from nodes that are offline and without it, you will get that error

How do you pin AD to a thread?

Fact you not aware of it, means its pinned. AD pins apps by default, so as long as you didn't manipulate it, you all good.

Sent you some coffee monies ❤️

Thanks and very much appreciated

[skynet01]

The name was never empty. I also have only 4 nodes, not 5, maybe it's not counting the nodes properly from the config file?

But even if that's the case, the state of a node should not be set as "undefined" it should always be an integer (ex 0) that way you can still have correct math when you calculate the confidence level for a person... or a better way is to exclude any "undefined" nodes from the calculation.

[skynet01]

@Odianosen25 any update on this? Is there a way to remove this extra node from my config? This unknown node is messing up presence calculations

[Odianosen25]

Hello @skynet01,

Ah didn’t even realise it was still an issue. Will add code to remove it, as it should have cleared and if it didn’t then it kind of makes more sense kind of in my head what might have happened.

Will push some updates tomorrow to fix this due to family stuffs, hope you will be available to test as I won’t be able to test it locally.

kind regards

[skynet01]

No problem, ill help you test it :)

[Odianosen25]

Hello @skynet01,

Quick question, did you reboot both AD and HA and the same entity with “None” was created?

[skynet01]

Yeah i rebooted it a few times. The none eventually shows up

[Odianosen25]

@skynet01 ,

Just working via this, I have updated dev to prevent this, but want to also update it to clean it out in the event it happens in future for some strange reasons. Can you confirm that you have a sensor in HA named something like "sensor.monitor_alex_none_conf"?

If you don't it will most likely mean its been cleaned in HA restarted but only AD having it, and will make more sense.

Regards

[skynet01]

@Odianosen25 Yeah i dont have that sensor. What's weird is only 3 sensors show up instead of 4.

[Odianosen25]

@skynet01,

I think you just unearthed a bug that was hidden all these while. I think it’s important to do some periodic cleaning of sensors, based on if location data is not received after a while based on the sensors available.

I will work on some solution this weekend, as it’s kind of not so straightforward.

[skynet01]

maybe it can be combined with this issue? :) #51

[Odianosen25]

Ok will see what design I can come up with.

[skynet01]

@Odianosen25 I might be wrong, but wouldn’t not allowing the monitors to never be set to “unknown “ would at least partially solve the issue?

[Odianosen25]

@skynet01, a sensor shouldn’t get to that level in the first place. No sensor is set to “unknown” as far as I know, as default is “0” for confidence. It’s an interesting bug, so looking at it

[skynet01]

Not sure if this is related but this might help you, i get this error in HA logs:

  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
    resp = await self._request_handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_app.py", line 499, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 77, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 24, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 144, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 135, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/api/__init__.py", line 270, in post
    hass.states.async_set(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1218, in async_set
    state = State(
  File "/usr/src/homeassistant/homeassistant/core.py", line 900, in __init__
    raise InvalidEntityFormatError(
homeassistant.exceptions.InvalidEntityFormatError: Invalid entity id encountered: sensor.monitor_vanessa_btsensor-bedroom_conf. Format should be <domain>.<object_id>

[skynet01]

Anyway I can manually delete this detected unknown monitor node? It shows up after all existing nodes report 0 confidence. So it’s not visible at first

[Odianosen25]

@skynet01,

Hope you good?

Apologies too a while, have a pregnant Mrs and comes with all the drama. Any ways do check the latest update and let me know.

It should attempt to clean when the app starts, and do it every hour. There is a short cut actually to remove it which I could have told you, but I think its better the app manages that for users, to improve on experience.

Kind regards

[skynet01]

No problem, i know how it is with all the family drama :) . Thanks for the update. I get the following error when the app starts though:

2021-07-15 17:12:53.494337 WARNING Error: ------------------------------------------------------------
2021-07-15 17:12:53.495367 WARNING Error: Unexpected error loading module: /config/appdaemon/apps/Monitor-App/home_presence_app.py:
2021-07-15 17:12:53.496480 WARNING Error: ------------------------------------------------------------
2021-07-15 17:12:53.505122 WARNING Error: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/appdaemon/app_management.py", line 963, in check_app_updates
    await utils.run_in_executor(self, self.read_app, mod["name"], mod["reload"])
  File "/usr/lib/python3.8/site-packages/appdaemon/utils.py", line 299, in run_in_executor
    response = future.result()
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.8/site-packages/appdaemon/app_management.py", line 755, in read_app
    self.modules[module_name] = importlib.import_module(module_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/config/appdaemon/apps/Monitor-App/home_presence_app.py", line 1661, in <module>
    {"mode":"full","isActive":false}
NameError: name 'false' is not defined

[Odianosen25]

Hello @skynet01,

That's an awkward error as there is no line 1661. Might have to do with the black formatting or something.

Can you just edit anything in the app so it reloads and see if it works?

Kind regards

[skynet01]

Hey @Odianosen25 you were right, reloading the app fixed it. I am still getting the None sensor, here is the start up log.

2021-07-22 10:19:41.671366 INFO AppDaemon: Initializing app home_presence_app using class HomePresenceApp from module home_presence_app
2021-07-22 10:19:41.739791 WARNING home_presence_app: home_presence_app: Entity monitor.monitor_state not found in namespace mqtt
2021-07-22 10:19:41.741918 INFO AppDaemon: home_presence_app: Entity monitor.monitor_state created in namespace: mqtt
2021-07-22 10:19:41.984441 INFO home_presence_app: Setting up Monitor auto reboot
2021-07-22 10:19:41.998382 INFO MQTT: Topic monitor/# already subscribed to
2021-07-22 10:19:42.059460 INFO AppDaemon: App initialization complete
2021-07-22 10:19:42.098287 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-22 10:19:43.077823 WARNING home_presence_app: Got an invalid location None, from topic monitor/echo
2021-07-22 10:19:43.127160 WARNING home_presence_app: home_presence_app: Entity monitor.corridor_state not found in namespace mqtt
2021-07-22 10:19:43.128086 INFO AppDaemon: home_presence_app: Entity monitor.corridor_state created in namespace: mqtt
2021-07-22 10:19:43.160056 WARNING home_presence_app: home_presence_app: Entity monitor.lounge_state not found in namespace mqtt
2021-07-22 10:19:43.161931 INFO AppDaemon: home_presence_app: Entity monitor.lounge_state created in namespace: mqtt
2021-07-22 10:19:43.203619 INFO AppDaemon: New client Admin Client connected
2021-07-22 10:19:43.494255 WARNING home_presence_app: home_presence_app: Entity monitor.btsensor-bedroom_state not found in namespace mqtt
2021-07-22 10:19:43.496251 INFO AppDaemon: home_presence_app: Entity monitor.btsensor-bedroom_state created in namespace: mqtt
2021-07-22 10:19:43.540401 WARNING home_presence_app: home_presence_app: Entity monitor.btsensor_state not found in namespace mqtt
2021-07-22 10:19:43.542273 INFO AppDaemon: home_presence_app: Entity monitor.btsensor_state created in namespace: mqtt
2021-07-22 10:19:43.576962 WARNING home_presence_app: Got an invalid location None, from topic monitor/state

[skynet01]

The good news is that it does appear like your clearing fix is working, i checked again just now and all the "None" stations are now gone. Ill report back if they come back.

[Odianosen25]

Hello @skynet01,

Oh that's nice to know, but I have found some other bug in the code, and will upload a new one over the weekend.

Kind regards

[skynet01]

Awesome, thanks. Is it regarding this? I am getting this every so often now:

2021-07-23 12:45:35.416374 WARNING home_presence_app: ------------------------------------------------------------
2021-07-23 12:45:35.418786 WARNING home_presence_app: Unexpected error in worker for App home_presence_app:
2021-07-23 12:45:35.420976 WARNING home_presence_app: Worker Ags: {'id': 'c96f226dac5d4c7a9b3ac7a05a29f933', 'name': 'home_presence_app', 'objectid': 'bc5418ad07c5447c9c463d6065b37aa3', 'type': 'event', 'event': 'MQTT_MESSAGE', 'function': <bound method HomePresenceApp.presence_message of <home_presence_app.HomePresenceApp object at 0xb533e748>>, 'data': {'topic': 'monitor/corridor/status', 'wildcard': 'monitor/#', 'payload': 'online'}, 'pin_app': True, 'pin_thread': 0, 'kwargs': {'wildcard': 'monitor/#', '__thread_id': 'thread-0'}}
2021-07-23 12:45:35.423014 WARNING home_presence_app: ------------------------------------------------------------
2021-07-23 12:45:35.425000 WARNING home_presence_app: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/appdaemon/threading.py", line 936, in worker
    funcref(args["event"], data, self.AD.events.sanitize_event_kwargs(app, args["kwargs"]))
  File "/config/appdaemon/apps/Monitor-App/home_presence_app.py", line 295, in presence_message
    self.handle_status(location=location, payload=payload.lower())
  File "/config/appdaemon/apps/Monitor-App/home_presence_app.py", line 495, in handle_status
    self.handle_nodes_state(location, payload)
  File "/config/appdaemon/apps/Monitor-App/home_presence_app.py", line 585, in handle_nodes_state
    attributes = self.mqtt.get_state(self.monitor_entity, attribute="all")[
TypeError: 'NoneType' object is not subscriptable
2021-07-23 12:45:35.427428 WARNING home_presence_app: ------------------------------------------------------------

[Odianosen25]

Hello @skynet01,

yes I believe it leads to that. I didn’t notice it until I saw the debug log you pasted last, so it was obvious to me some stuffs will start going wrong.

thanks for all the logs, should fix it tomorrow morning.

kind regards

[Odianosen25]

@skynet01,

Please try it now.

Kind regards

[skynet01]

@Odianosen25 that error is gone, but the underlying issue still remains. Here is the example screenshot. Vanessa is marked as away even though btsensor_bedroom is reporting 100% confidence.

021-07-26 17:08:06.987770 INFO AppDaemon: home_presence_app: Entity sensor.monitor_family_tania_btsensor-bedroom_conf created in namespace: default
2021-07-26 17:08:07.012868 WARNING HASS: Error setting Home Assistant state default.sensor.monitor_family_tania_btsensor-bedroom_conf, {'state': 100, 'attributes': {'friendly_name': 'Family Tania Btsensor-Bedroom Confidence', 'unit_of_measurement': '%'}}
2021-07-26 17:08:07.014190 WARNING HASS: Code: 500, error: 500 Internal Server Error
Server got itself in trouble
2021-07-26 17:08:07.020704 ERROR home_presence_app: Entity sensor.monitor_family_tania_btsensor-bedroom_conf does not exist, running arrival scan.
2021-07-26 17:08:07.074277 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:09.828059 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:11.471322 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:14.798582 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:16.077052 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:25.693296 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:29.728340 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:37.166438 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:37.205861 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:45.316710 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:45.435993 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:08:53.931428 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:09:01.218332 WARNING home_presence_app: home_presence_app: Entity monitor.vanessa_None not found in namespace mqtt
2021-07-26 17:09:01.221602 INFO AppDaemon: home_presence_app: Entity monitor.vanessa_None created in namespace: mqtt
2021-07-26 17:09:01.227458 ERROR home_presence_app: Entity sensor.monitor_vanessa_btsensor-bedroom_conf does not exist, running arrival scan.
2021-07-26 17:09:02.785203 WARNING home_presence_app: Got an invalid location None, from topic monitor/state
2021-07-26 17:09:07.426622 WARNING home_presence_app: Got an invalid location None, from topic monitor/state

[Odianosen25]

@skynet01,

Hope you are good? Been a while and apologies for my silence all these while, been a wild ride. New job, baby coming, looking for a home and stuffs.

I have made a couple of changes to the code, and kindly try out the new release. It should fix some of the issues you have experienced.

Kind regards

[skynet01]

Congrats on the baby coming @Odianosen25 ! I figured out that the issue was linked to naming of my sensor. Apparently having a "-" in a name btsensor-bedroom created all the issues with undefined. Removing the dash and just having the name bedroom fixed the issues for me.

[Odianosen25]

Ah that’s good catch @skynet01 and thanks. Do u think there is anything the app can do to help others that could make the same mistake? Or it could just be made clear with a big “don’t do”?

[skynet01]

its probably easier in the config to mention not to use special characters in the name. Looks like it's pretty rare occurrence

[Odianosen25]

Ok cool thanks and will do that