Closed gribber closed 1 year ago
Could you please try latest beta 1.6.0 of powercalc? A lot of things has been changed to improve the group energy sensors. Hope this also solves your issue. Please create a backup of your HA first just in case.
Btw these errors in your log are definitely related:
ValueError: Sensor sensor.kitchen_power has device class power, state class measurement unit W and suggested precision None thus indicating it has a numeric value; however, it has the non-numeric value: unavailable (<class 'str'>)
2023-05-08 08:40:49.740 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up powercalc platform for sensor
There are a lot of integrations which are broken or misbehaving due to this change.
v1.6.0 of powercalc should also handle this case, but there are a lot of things changed in this version, mostly related to the way group sensors energy is calculated. See #796 for more information. I did not get any response from any user which has tested this version of powercalc and would really need some confirmations before releasing this to the public.
Could you please try latest beta 1.6.0 of powercalc? A lot of things has been changed to improve the group energy sensors. Hope this also solves your issue. Please create a backup of your HA first just in case.
Sure, I can try it out, what is the best way to switch to beta version?
In hacs you can click redownload and mark the check box with beta versions.
I tried the latest beta.9. I still doesnt see any energy value for the light group defined in config.yaml by "create_domain_group" , only unavailable. The power entity (W) works but not the energy (kWh) .
Updated to beta.9 Same result here, energy still unavailable
log:
@gribber At least all the errors disappeared now, which is promising.
The clue is here:
2023-05-08 12:44:20.490 WARNING (MainThread) [custom_components.powercalc.sensors.group] sensor.living_room_energy: One or more members of the group are unavailable, setting group to unavailable (sensor.livingroom_front_3_energy,sensor.mushroom_lamp_energy,sensor.livingroom_back_3_energy,sensor.livingroom_back_1_energy,sensor.livingroom_window_1_energy,sensor.livingroom_front_2_energy,sensor.livingroom_back_2_energy,sensor.livingroom_front_4_energy)
It's a precaution I added to avoid energy sensor getting negative. It will check if there is a previous reading of this sensor (before it went unavailable). When it doesn't it will raise this warning. Will need to dig in this some more to see if this is really necessary or I could just ignore this. Has been a few weeks ago I added this code.
Could you try making them available one time and see if the energy sensor starts working after that?
I tried the latest beta.9. I still doesnt see any energy value for the light group defined in config.yaml by "create_domain_group" , only unavailable. The power entity (W) works but not the energy (kWh) .
Could you have a look into the logs if you have similar warning as gribber? One or more members of the group are unavailable
livingroom_front_3_energy
Yep, it came back again when they all are available for that area, statistics still messed up but I guess I have to update the database to make them correct again
The question is how to handle this further on, lamps will continue to move to other rooms, get connected/disconnected by season and replaced when broken
For groups, in my opinion, statistics should remain what ever it was before entities got disconnected/moved and not reduced by the amount of energy that entity already consumed
On the other hand, I don't want the new group to grow by the same amount energy already used by a entity if it's moved
The entity itself, unless deleted/recreated should keep whatever stats it has regardless of area
Feels a bit tricky to solve, more work than just sum up entities in a area
Previously it would just sum up every entity in the group.
In the new implementation v1.6.0 it will calculate a delta for each entity.
It saves previous entity state of individual entities in config/.storage/powercalc_group
.
When one of the group entities changes state it will lookup the last state there and calculate the delta, so the group sensor only increases. Regardless of unavailable entities or removal/addition of entities in the group.
It should handle the cases you mentioned.
But it's hard to get right and to forsee all the edge cases.
Previously it would just sum up every entity in the group.
In the new implementation v1.6.0 it will calculate a delta for each entity. It saves previous entity state of individual entities in
config/.storage/powercalc_group
. When one of the group entities changes state it will lookup the last state there and calculate the delta, so the group sensor only increases. Regardless of unavailable entities or removal/addition of entities in the group.It should handle the cases you mentioned.
But it's hard to get right and to forsee all the edge cases.
Oh I see, if I understand that correct, it should work as I want :-)
So what's left is groups with unavailable lights still has state unavailable (and groups which included groups with that problem)
got lots of these now:
2023-05-08 14:11:53.268 DEBUG (MainThread) [custom_components.powercalc.sensors.group] delta for entity sensor.livingroom_back_1_energy: -0.001399999999999999985775267497
2023-05-08 14:11:53.268 WARNING (MainThread) [custom_components.powercalc.sensors.group] skipping state for sensor.livingroom_back_1_energy, probably erroneous value or sensor was reset
2023-05-08 14:11:53.268 DEBUG (MainThread) [custom_components.powercalc.sensors.group] sensor.living_room_energy: New value: 1.810500000000000000000000000
2023-05-08 14:11:53.269 WARNING (MainThread) [custom_components.powercalc.sensors.group] sensor.indoor_lights_energy: One or more members of the group are unavailable, setting group to unavailable (sensor.kitchen_lamp_energy,sensor.kitchen_table_energy,sensor.kitchen_window_1_energy,sensor.laundry_room_window_energy)
2023-05-08 14:11:53.270 DEBUG (MainThread) [custom_components.powercalc.strategy.lut] light.livingroom_back_2: Looking up power usage for bri:1 mired:328
2023-05-08 14:11:53.270 DEBUG (MainThread) [custom_components.powercalc.strategy.lut] light.livingroom_back_2: Calculated power:0.64000000000000001332267629550187848508358001708984375
2023-05-08 14:11:53.270 DEBUG (MainThread) [custom_components.powercalc.sensors.power] light.livingroom_back_2: State changed to "on". Power:0.64
2023-05-08 14:11:53.270 DEBUG (MainThread) [custom_components.powercalc.strategy.lut] light.livingroom_back_3: Looking up power usage for bri:1 mired:328
2023-05-08 14:11:53.270 DEBUG (MainThread) [custom_components.powercalc.strategy.lut] light.livingroom_back_3: Calculated power:0.64000000000000001332267629550187848508358001708984375
2023-05-08 14:11:53.270 DEBUG (MainThread) [custom_components.powercalc.sensors.power] light.livingroom_back_3: State changed to "on". Power:0.64
2023-05-08 14:11:53.270 DEBUG (MainThread) [custom_components.powercalc.strategy.lut] light.livingroom_back_4: Looking up power usage for bri:1 mired:328
2023-05-08 14:11:53.270 DEBUG (MainThread) [custom_components.powercalc.strategy.lut] light.livingroom_back_4: Calculated power:0.64000000000000001332267629550187848508358001708984375
2023-05-08 14:11:53.270 DEBUG (MainThread) [custom_components.powercalc.sensors.power] light.livingroom_back_4: State changed to "on". Power:0.64
2023-05-08 14:11:53.275 DEBUG (MainThread) [custom_components.powercalc.sensors.group] sensor.living_room_energy: Recalculate, current value: 1.8105
2023-05-08 14:11:53.275 DEBUG (MainThread) [custom_components.powercalc.sensors.group] delta for entity sensor.livingroom_back_2_energy: -0.001399999999999999985775267497
2023-05-08 14:11:53.275 WARNING (MainThread) [custom_components.powercalc.sensors.group] skipping state for sensor.livingroom_back_2_energy, probably erroneous value or sensor was reset
2023-05-08 14:11:53.276 DEBUG (MainThread) [custom_components.powercalc.sensors.group] delta for entity sensor.livingroom_back_4_energy: -0.001399999999999999985775267497
2023-05-08 14:11:53.276 WARNING (MainThread) [custom_components.powercalc.sensors.group] skipping state for sensor.livingroom_back_4_energy, probably erroneous value or sensor was reset
2023-05-08 14:11:53.277 DEBUG (MainThread) [custom_components.powercalc.sensors.group] delta for entity sensor.livingroom_back_3_energy: -0.001399999999999999985775267497
2023-05-08 14:11:53.277 WARNING (MainThread) [custom_components.powercalc.sensors.group] skipping state for sensor.livingroom_back_3_energy, probably erroneous value or sensor was reset
2023-05-08 14:11:53.277 DEBUG (MainThread) [custom_components.powercalc.sensors.group] delta for entity sensor.floor_lamp_energy: 0E-61
every entity seems to have messed up (negative) values now, the statistics can be fixed by the tool in developer tools, but the state I guess have to be fixed by a database update either way, it's above 100 entities 😓
Did you have messed up values before upgrade to v1.6.0-beta or after upgrade?
Regarding:
delta for entity sensor.livingroom_back_2_energy: -0.001399999999999999985775267497
It's comparing the current state of sensor.livingroom_back_2_energy
to the last state saved in config/.storage/powercalc_group
. When this is negative I just throw a warning so the group value won't go negative.
Not sure how the current value of this sensor can be lower than the previous. Might need some more debug logging to find tha`t out.
Before, but not at the 2023.5.0 update, this morning for some reason, at 8.50, almost every entity got reset except this: https://github.com/gribber/ha/blob/6c70c9f2a654a082f69f3ba8c8297dcd3913cdef/sensors/powercalc.yaml#L160-L179 which is all fine and got correct values and this: https://github.com/gribber/ha/blob/6c70c9f2a654a082f69f3ba8c8297dcd3913cdef/sensors/powercalc.yaml#L54-L58 which got was fine, then got reset at 14:05 (a little more than 1h after update)
I'm confused
Before, but not at the 2023.5.0 update, this morning for some reason, at 8.50, almost every entity got reset except this: https://github.com/gribber/ha/blob/6c70c9f2a654a082f69f3ba8c8297dcd3913cdef/sensors/powercalc.yaml#L160-L179 which is all fine and got correct values and this: https://github.com/gribber/ha/blob/6c70c9f2a654a082f69f3ba8c8297dcd3913cdef/sensors/powercalc.yaml#L54-L58 which got was fine, then got reset at 14:05 (a little more than 1h after update)
I'm confused
It does say this:
2023-05-08 14:09:33.664 DEBUG (MainThread) [custom_components.powercalc.sensors.abstract] Binding sensor.bathroom_downstairs_mirror_energy to device a2282cdb5f629eaf14e8bcd59a1fc166
2023-05-08 14:09:33.664 WARNING (MainThread) [homeassistant.components.integration.sensor] sensor.bathroom_downstairs_mirror_energy could not restore last state unavailable: [<class 'decimal.ConversionSyntax'>]
2023-05-08 14:09:33.667 DEBUG (MainThread) [custom_components.powercalc.sensors.group] sensor.bathroom_energy: Restoring state: 16.0513
so it seems to fail to restore in some way
the group in which this light exist (this is the only one is this group)
2023-05-08 14:19:33.252 DEBUG (MainThread) [custom_components.powercalc.sensors.group] sensor.bathroom_energy: Recalculate, current value: 16.0513
2023-05-08 14:19:33.252 DEBUG (MainThread) [custom_components.powercalc.sensors.group] delta for entity sensor.bathroom_downstairs_mirror_energy: -16.05120000000000100043458760
2023-05-08 14:19:33.252 WARNING (MainThread) [custom_components.powercalc.sensors.group] skipping state for sensor.bathroom_downstairs_mirror_energy, probably erroneous value or sensor was reset
2023-05-08 14:19:33.252 DEBUG (MainThread) [custom_components.powercalc.sensors.group] sensor.bathroom_energy: New value: 16.0513
the state value of this group is correct and has not been reset
attaching the logs
Hard to say why your sensors gets reset all of a sudden in the past, I have no clue, sorry.
2023-05-08 14:09:33.664 WARNING (MainThread) [homeassistant.components.integration.sensor] sensor.bathroom_downstairs_mirror_energy could not restore last state unavailable: [<class 'decimal.ConversionSyntax'>]
This is actually a problem of the home assistant riemann integral component.
Which bugs on unavailable
state. This should be reported at HA issue tracker.
But this is not a problem per se. On startup it tries to set the energy sensor to the last known state (which was unavailable
), and it stays unavailable so that's not really an issue. Also it's only a warning.
What happens if you make the sensor sensor. bathroom_downstairs_mirror_energy
available again.
Does it start adding values to the group again?
And does it turn this line positive again?
2023-05-08 14:19:33.252 DEBUG (MainThread) [custom_components.powercalc.sensors.group] delta for entity sensor.bathroom_downstairs_mirror_energy: -16.05120000000000100043458760
Hard to say why your sensors gets reset all of a sudden in the past, I have no clue, sorry.
2023-05-08 14:09:33.664 WARNING (MainThread) [homeassistant.components.integration.sensor] sensor.bathroom_downstairs_mirror_energy could not restore last state unavailable: [<class 'decimal.ConversionSyntax'>]
This is actually a problem of the home assistant riemann integral component. Which bugs on
unavailable
state. This should be reported at HA issue tracker. But this is not a problem per se. On startup it tries to set the energy sensor to the last known state (which wasunavailable
), and it stays unavailable so that's not really an issue. Also it's only a warning.What happens if you make the sensor
sensor. bathroom_downstairs_mirror_energy
available again. Does it start adding values to the group again? And does it turn this line positive again?
2023-05-08 14:19:33.252 DEBUG (MainThread) [custom_components.powercalc.sensors.group] delta for entity sensor.bathroom_downstairs_mirror_energy: -16.05120000000000100043458760
This is what happens if I turn that light on
2023-05-08 16:15:21.436 DEBUG (MainThread) [custom_components.powercalc.sensors.power] light.bathroom_downstairs_mirror: State changed to "on". Power:12.30
2023-05-08 16:15:21.439 DEBUG (MainThread) [custom_components.powercalc.sensors.group] sensor.bathroom_energy: Recalculate, current value: 16.0570
2023-05-08 16:15:21.439 DEBUG (MainThread) [custom_components.powercalc.sensors.group] delta for entity sensor.bathroom_downstairs_mirror_energy: 0.0001000000000000002622901895677
2023-05-08 16:15:21.439 DEBUG (MainThread) [custom_components.powercalc.sensors.group] sensor.bathroom_energy: New value: 16.05710000000000000026229019
2023-05-08 16:15:21.439 WARNING (MainThread) [custom_components.powercalc.sensors.group] sensor.indoor_lights_energy: One or more members of the group are unavailable, setting group to unavailable (sensor.kitchen_lamp_energy,sensor.kitchen_table_energy,sensor.kitchen_window_1_energy,sensor.laundry_room_window_energy)
2023-05-08 16:15:31.515 DEBUG (MainThread) [custom_components.powercalc.sensors.power] light.bathroom_downstairs_mirror: State changed to "off". Power:1.20
Nothing else in the logs regarding this
Both sensors (the light and the group) continues to count
That increase of energy sensor looks good.
I will have a look into One or more members of the group are unavailable
.
Have an idea how to improve upon that.
It should not make the group unavailable when only a few sensors are unavailable. But just skip these from adding delta values to the energy sensor.
Will probably have a look tonight and issue a new beta.
Just released a new beta which should hopefully also resolve the unavailable issue for the energy group for you. v1.6.0-beta.10
Just released a new beta which should hopefully also resolve the unavailable issue for the energy group for you. v1.6.0-beta.10
Solved the problem here after updating to v1.6.0-beta.10
It has appears other errors, but I need to investigate the origin if it is related or not.
2023-05-08 17:57:08.721 ERROR (MainThread) [custom_components.powercalc.helpers] Could not convert power value float(0) to decimal
@daniperaleda I did have a look and this error is raised when the fixed power strategy returns 0 as the power. However that should just be a legit case and not cause an error. I will have a look into this.
I don't think it will break anything or cause malfunction though as things are also handled later on.
@daniperaleda I cannot reproduce this. Could you please create a new issue for this and add your configuration of fixed power sensors. Do you use some template which is returning "float(0)" literaly? We'll discuss further in other issue to not hijack this one.
@daniperaleda I cannot reproduce this. Could you please create a new issue for this and add your configuration of fixed power sensors. Do you use some template which is returning "float(0)" literaly? We'll discuss further in other issue to not hijack this one.
FIXED...I have reviewed the configuration...it was a TEMPLATE POWER CALCULATE sensor...and it has a "{" missing...so it got a literal not a number....so OK for everything.
FIXED...I have reviewed the configuration...it was a TEMPLATE POWER CALCULATE sensor...and it has a "{" missing...so it got a literal not a number....so OK for everything.
Cool glad you got that resolved. will add some validations on correct template #1666.
I can confirm it work here as well
Is there a way to manually update the values of each energy sensor, at least the groups, which doesn't mess things up?
Is there a way to manually update the values of each energy sensor, at least the groups, which doesn't mess things up?
I'm afraid not.
Will need to do some DB queries to correct things. Because statistics are calculated by looking at previous statistic. When it is wrong at a certain point it won't be corrected.
Do you use MariaDB or SQLite?
You can have a look into statistics
, statistics_short_term
and states
tables for one problematic entity and see where the problem is, where the data got "corrupted".
Maybe I can help suggest a way to correct that on your HA installation.
Is there a way to manually update the values of each energy sensor, at least the groups, which doesn't mess things up?
I'm afraid not. Will need to do some DB queries to correct things. Because statistics are calculated by looking at previous statistic. When it is wrong at a certain point it won't be corrected. Do you use MariaDB or SQLite? You can have a look into
statistics
,statistics_short_term
andstates
tables for one problematic entity and see where the problem is, where the data got "corrupted". Maybe I can help suggest a way to correct that on your HA installation.
Using mariadb, the statistics can be fairly easy corrected in the gui (dev tools - > statistics), at least I think that's enough, so it's the state that needs update The db layout seems to have changed since last time I had to some something similar
Closing this issue as there were also fixes in HA core latest versions regarding energy sensors misbehaving (going negative and things like that).
System Health details
System Information
Home Assistant Community Store
GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 4632 Installed Version | 1.32.1 Stage | running Available Repositories | 1284 Downloaded Repositories | 50Home Assistant Cloud
logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | okHome Assistant Supervisor
host_os | Debian GNU/Linux 11 (bullseye) -- | -- update_channel | stable supervisor_version | supervisor-2023.04.1 agent_version | 1.2.1 docker_version | 20.10.21 disk_total | 78.2 GB disk_used | 55.7 GB healthy | true supported | true supervisor_api | ok version_api | ok installed_addons | Studio Code Server (5.5.7), MariaDB (2.6.0), InfluxDB (4.6.0), Mosquitto broker (6.2.1), phpMyAdmin (0.8.5), ESPHome (2023.4.4), Frigate NVR Proxy (1.3), Node-RED (14.1.5), Zigbee2MQTT (1.30.4-1), Network UPS Tools (0.11.2)Dashboards
dashboards | 2 -- | -- resources | 30 views | 20 mode | storageRecorder
oldest_recorder_run | May 4, 2023 at 04:29 -- | -- current_recorder_run | May 8, 2023 at 08:47 estimated_db_size | 1681.28 MiB database_engine | mysql database_version | 10.6.12Spotify
api_endpoint_reachable | ok -- | --Checklist
Describe the issue
Noticed today that all energy counters got reset, seems all groups are marked as unavailable until at least one lamp is turned on/off after reboot of HA
I have nestled groups, like "indoor lights -> roomX", like this: https://github.com/gribber/ha/blob/main/sensors/powercalc.yaml
every room which had at least one unavailable lamp is marked as unavailable (for example, I have some christmas lamps disconnected) Turing on any other lamp in that room makes the group energy counter available again, but with incorrect values
living room group
main indoor light group as seen, this groups hasn't changed since 06:24:55, which is exactly the time I updated to HA 2023.5.0
Reproduction steps
something like:
Debug logs
Diagnostics dump or YAML config
platform: powercalc create_group: "outdoor lights" unique_id: 28782826-4b21-44d0-8105-60e6b9338bd2 entities:
platform: powercalc create_group: "indoor lights" unique_id: 0259ae80-3fd3-40fe-9cf9-1857f2e28fc0 entities:
platform: powercalc create_group: "garage lights" unique_id: 765c1684-143e-43da-80b6-8f0ca239db38 entities:
platform: powercalc unique_id: 3788b38d-5330-4c7e-a0ea-8565dda2c6c9 entity_id: sensor.dummy name: "techups" fixed: power: "{{states('sensor.techups_power_tmp')}}"
platform: powercalc create_group: "smart speakers" unique_id: f883310e-e09c-4d1c-8ef7-b2e83b530324 entities:
platform: powercalc create_group: "environment and air quality" unique_id: f5f210d6-1608-467e-b272-a7b0a83ae6b2 entities: