Energy sensors became unavailable

vahaldor commented 1 year ago

System Health details

System Information

version	core-2023.5.4
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.10.11
os_name	Linux
os_version	6.1.29
arch	x86_64
timezone	Europe/Vienna
config_dir	/config

Home Assistant Community Store

Home Assistant Cloud

Home Assistant Supervisor

Dashboards

Recorder

Sonoff

version | 3.5.1 (122751c) -- | -- cloud_online | 2 / 2 local_online | 0 / 0

Checklist

[X] I have read the FAQ to see if there is a known solution to my problem.
[X] I have enabled debug logging for my installation.
[X] I have filled out the issue template to the best of my ability.
[X] This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
[X] This issue is not a duplicate issue of currently previous issues..

Describe the issue

Suddenly all energy sensors in powercal became unavailable. I updated to the newest version 1.6.3, which seems to address something similar.

Reproduction steps

...

Debug logs

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 304, in _async_setup_platform
    await asyncio.shield(task)
  File "/config/custom_components/powercalc/sensor.py", line 268, in async_setup_entry
    entities = await create_group_sensors_from_config_entry(
  File "/config/custom_components/powercalc/sensors/group.py", line 187, in create_group_sensors_from_config_entry
    resolve_entity_ids_recursively(hass, entry, SensorDeviceClass.ENERGY),
  File "/config/custom_components/powercalc/sensors/group.py", line 352, in resolve_entity_ids_recursively
    resolved_ids.extend([sorted_entities[0]])
IndexError: list index out of range

Diagnostics dump or YAML config

No response

vahaldor commented 1 year ago

UPDATE: All energy sensors are suddenly back but the group of other powercalc sensors remains unavailable

vahaldor commented 1 year ago

Recreating the group does not produce entities.

bramstroker commented 1 year ago

Strange error you got here IndexError: list index out of range. Could you also enable debug logs please and restart with 1.6.2 / 1.6.3 to see if that provides any insights? You can attach the logs here or sent me private at bgerritsen_at_gmail_dot_com

It should not be necessary to recreate the group.

Could you also share diagnostics dump for the group entry which fails. You can click the three dots from the integration panel and then download diagnostics.

vahaldor commented 1 year ago

1) This is the only log using keyword "powercalc" in settings/system/logs

Logger: homeassistant.components.sensor Source: custom_components/powercalc/sensors/group.py:352 Integration: Sensor (documentation, issues) First occurred: 18:13:44 (3 occurrences) Last logged: 18:13:46

Error while setting up powercalc platform for sensor Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 304, in _async_setup_platform await asyncio.shield(task) File "/config/custom_components/powercalc/sensor.py", line 268, in async_setup_entry entities = await create_group_sensors_from_config_entry( File "/config/custom_components/powercalc/sensors/group.py", line 187, in create_group_sensors_from_config_entry resolve_entity_ids_recursively(hass, entry, SensorDeviceClass.ENERGY), File "/config/custom_components/powercalc/sensors/group.py", line 352, in resolve_entity_ids_recursively resolved_ids.extend([sorted_entities[0]]) IndexError: list index out of range

2) Diagnostics attached config_entry-powercalc-e3bd28a393cd90c0465d7356ac13679d.json (2).txt

bramstroker commented 1 year ago

@vahaldor Thanks for the logs and diagnostics. You should have a lot more log lines (hundreds) after you enable debug logging for powercalc. see https://homeassistant-powercalc.readthedocs.io/en/latest/troubleshooting/debug-logging.html Do you know in which powercalc version this was still working correctly. Hopefully we can pinpoint the exact change which caused this more easily?

bramstroker commented 1 year ago

It's a problem with one of the underlying member entities of this group. For one (or maybe multiple of them) no power and/or energy sensors are created and the group creation logic fails on that. I have made a change so no fatal exception is thrown, but the group creation just continues and the problematic sensor is logged as an error. This change is in #1687. Will be merged into master within a few minutes. Are you able to test this by installing the master branch? You can do that with HACS, click redownload and enable beta versions in this process.

vahaldor commented 1 year ago

Installed 1.6.4., no change Installed master (after enabling beta), no change

Logs attached home-assistant_2023-05-30T18-16-26.130Z.log

vahaldor commented 1 year ago

I checked all lights in the group, they seem to be all available. There are no other unavailable entities than the group from powercalc

bramstroker commented 1 year ago

I have checked the logs but I still see the following error:

2023-05-30 20:14:21.681 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up powercalc platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 304, in _async_setup_platform
    await asyncio.shield(task)
  File "/config/custom_components/powercalc/sensor.py", line 269, in async_setup_entry
    entities = await create_group_sensors_from_config_entry(
  File "/config/custom_components/powercalc/sensors/group.py", line 187, in create_group_sensors_from_config_entry
    resolve_entity_ids_recursively(hass, entry, SensorDeviceClass.ENERGY),
  File "/config/custom_components/powercalc/sensors/group.py", line 352, in resolve_entity_ids_recursively
    resolved_ids.extend([sorted_entities[0]])
IndexError: list index out of range

I'm 99% confident this error should not be in the logs anymore when you are on latest master.

Could you check the version displayed at the integration page in HACS?

Version which should be displayed there is 98b07a11

bramstroker commented 1 year ago

You could also check all the entries in the GUI configuration. They all should have at least 1 entity. I suspect there is one without any entities

vahaldor commented 1 year ago

I am running the version 1.6.4 and all the energy sensors have been working, I did not change anything. This problem emerged today around 9AM from nowhere, I did not update powercalc. I checked the update after I found that the group is unavilable because then all other sensor (all flat) I calculate later in configuration.yaml became unavailable.

I checked all entries of all lights and all of them have entities.

bramstroker commented 1 year ago

I don't understand exactly. It's solved for you now or you still have an issue?

If solved I have no clue why things suddenly broke for you. Have never seen this issue or reported by anyone. At least I have added some extra failsafe now in the master branch (will be release in v1.6.5) which makes sure the group sensor will still be created when one of the member sensors has an setup issue.

vahaldor commented 1 year ago

No, it is not solved. My group is still unavailable. All other configurations, out of which the group is created, contain entities, so I am not sure whether this update can cause any difference.

What next? :)

bramstroker commented 1 year ago

Well, if I understand correctly you are still running 1.6.4, which don't contains a fix for the error you experiencing. So you must really install the master branch, not 1.6.4. You can select that when clicking redownload in HACS.

vahaldor commented 1 year ago

I redownloaded the integratin, chose "master" when beta channel is turned on. Restarted HA, no difference and the integration still shows version 1.6.4

bramstroker commented 1 year ago

That's strange. I can remember I had that issue also once. Maybe try another time redownload in the hope it will finally download the right version. There should be some code (commit hash) shown in the version at the left top, not 1.6.4

bramstroker commented 1 year ago

Just tried on my test machine, also needed to trigger download 3 times. It was very slow. But eventually it finished. You should see this version. Before restarting just after download you should already see this.

vahaldor commented 1 year ago

ok, solved! It was LIGHT | Nikola point. This light lost its energy sensor. So I recreated it. It is one of the simplest light as it is a lamp turned on by TAPO plug, so fixed power consumption. Why it was gone is beyond me. Thank you for insisting where was the problem!

However, I appreciate your willingness to keep sensor calculating while one sensor became unavailable. This is massive advantage now for powercalc, because the feature that my flat consumption is not calculated when one sensor fails using templates in configuration.yaml is really annoying. Thanks!

Now I can probably create every calculations in powercalc. Keep doing great job!

bramstroker commented 1 year ago

ok, solved! It was LIGHT | Nikola point. This light lost its energy sensor. So I recreated it. It is one of the simplest light as it is a lamp turned on by TAPO plug, so fixed power consumption. Why it was gone is beyond me. Thank you for insisting where was the problem!

Ah good to hear it's resolved for you. Strange that the sensor was not created anymore. The logs should definitely give some insights on that kind of issues. Also I saw you have some warnings in the logs. Not all specifically related to powercalc. Would be a good idea to solve all warnings also to prevent issues in the future. Ideally you should not have any errors or warnings in the logs.

However, I appreciate your willingness to keep sensor calculating while one sensor became unavailable. This is massive advantage now for powercalc, because the feature that my flat consumption is not calculated when one sensor fails using templates in configuration.yaml is really annoying. Thanks!

Now I can probably create every calculations in powercalc. Keep doing great job!

Glad you like it.

viking2010 commented 1 year ago

I've had the same issue over the last week with powercalc energy sensors going unavailable. I'm using the latest version and have the energy sensors setup in my sensors.yaml file. If I turn the device on, the sensors come back online again.

I'll try and get some debug info for you tomorrow when I get home.

bramstroker commented 1 year ago

@viking2010 Do you still experience issues? If so could you provide debugging information? Could also be an issue in HA core, last month there were multiple reports and fixes due to misbehaving energy sensors going negative or unavailable. So I'm not sure if it's an issue I can fix in Powercalc. But happy to have a look if you can provide additional information / logs after startup etc.

Joao-Sousa-71 commented 1 year ago

I've had the same issue over the last week with powercalc energy sensors going unavailable. I'm using the latest version and have the energy sensors setup in my sensors.yaml file. If I turn the device on, the sensors come back online again.

I'll try and get some debug info for you tomorrow when I get home.

I updated today from the 2023.4.6 to 2023.5.4 and have the same issue. My virtual power sensors (fixed) are unavailable when home Assistant restart until are switched on or after a few minutes but they lost the cumulated energy value since is reset to 0 kWh. I’m using Powecalc 1.6.6 and never notice this prior to the upgrade. I will do an upgrade to 2023.6.1 as soon is available for docker installations. And let you know.

Joao-Sousa-71 commented 1 year ago

It took me severall hous but already found what happened and a valid workaround. After the upgrade to the version 2023.5.4 almost all the virtual fixed energy sensors were unavailable for some time and others would not even became available. This is an example of one sensor sensor.sonoff_1000d7b012_active which returns "1" when switched on and "0" for all others states. The powercalc configuration was:

`- platform: powercalc

entity_id: sensor.sonoff_1000d7b012_active # Luz Cozinha fixed: states_power: 1: 17.9 # Potencia Lampada Led Cozinha
0: 0 ` With this, had this behaviour, the red circles show the sensor unavailability and match every restart in Home Assistant:

After changing the powercalc to this config and provide a value for the "0" state (btw 0.6 is the power of the sonoff when connected and with the relay off)

`- platform: powercalc

entity_id: sensor.sonoff_1000d7b012_active # Luz Cozinha fixed: states_power: 1: 17.9 # Potencia Lampada Led Cozinha
0: 0.6 ` The issue disappeared...

@bramstroker not sure if this information is enough for you, however it seems not hard to replicate. If you prefer i can open a new issue. It seems that something has changed in the latest versions of HA, but hope this can help anybody else that can face the same. Thank you.

bramstroker commented 1 year ago

@Joao-Sousa-71 Thanks for the information. Please have a look into https://github.com/home-assistant/core/issues/92912 as this seems the same issue as you have and a lot of people reported this issue. Did you upgrade already to latest HA version, as a fix should already be available.

Joao-Sousa-71 commented 1 year ago

@Joao-Sousa-71 Thanks for the information.

Welcome. I'm thankfull for your support 👍

Did you upgrade already to latest HA version

Not yet. I plan do it over the weeked. Versao 2023.6.2 appeared today for docker. I'll let you know. Thank you again. Joao

Joao-Sousa-71 commented 1 year ago

Did you upgrade already to latest HA version, as a fix should already be available.

@bramstroker I did the update for the version 2023.6.3 last weekend and it didn't happen again. Also, sometimes some sensors started to have a crazy comsumption (suddenly very high) and it seems that is also fixed. Thank you.

bramstroker commented 1 year ago

Great. Closing this issue.

viking2010 commented 1 year ago

@viking2010 Do you still experience issues? If so could you provide debugging information? Could also be an issue in HA core, last month there were multiple reports and fixes due to misbehaving energy sensors going negative or unavailable. So I'm not sure if it's an issue I can fix in Powercalc. But happy to have a look if you can provide additional information / logs after startup etc.

@bramstroker Apologies, for some reason my alerts aren't working from GitHub so have only just seen this message. I am still having this issue. The PowerCalc sensors for Total and Daily energy used are still going unavailable. I've checked if any of the devices themselves are becoming uncontactable, but my logs show they're not dropping packets.

I haven't setup anything using the PowerCalc UI, as I'm still using the YAML setup.

powercalc:
  force_update_frequency: 00:10:00
  create_energy_sensors: true
  create_utility_meters: true
  utility_meter_types: 
    - daily
    - weekly
    - monthly
  enable_autodiscovery: false
  energy_sensor_precision: 4
  ignore_unavailable_state: true

logger:
  default: warning
  logs:
    custom_components.powercalc: debug

I then have multiple sensors setup like this:

  - platform: powercalc
    entity_id: switch.tumble_dryer
    power_sensor_id: sensor.tumble_dryer_power

Life overtook me when I last messaged on here and didn't get the debug logs sorted, so I've sorted them now, and will post again when I have some content in the logs.

UPDATE#1: I did forget to put in the above that after a reboot of HA, the total/daily energy sensors do appear. But, they drop off after a period of time. Logging is now on and I can see debug logs growing. When I get a sensor drop, which could be in the next 24hrs, I'll update this issue again.

viking2010 commented 1 year ago

Here's the Debug log from the last couple of days - hopefully this will help, but I'm not seeing a lot in there that means much to me.

Debug Log - PowerCalc.txt

bramstroker commented 1 year ago

@viking2010 Which sensors are getting unavailable exactly? You are sure you are on the latest version of HA now? And recent version of Powercalc.

The energy sensors and utility meters, which I guess you have problems with, are actually provided by HA core integration. Powercalc only has some magic to create them automatically. So the bug would be also there and probably be better to report at HA core issue tracker.

To rule out that powercalc is the problem you can also just create the energy sensor and utility meter using the HA helpers section (Devices&Services > Helpers). create a Riemann integral sensor (energy sensor) and a Utility meter. When these also have the same problem it's 100% certain it's not a powercalc issue.

You could enable debug logging for those components, maybe that gives more insights.

logger:
  default: warning
  logs:
    custom_components.powercalc: debug
    homeassistant.components.utility_meter: debug
    homeassistant.components.integration: debug

viking2010 commented 1 year ago

Will give that a go and feedback. The sensors going unavailable never seem to be listed within the Debug logs, which also makes me think it's something at another level that's the issue. They are all either smart lights or smart plugs (Using Local Tuya). As for the versions, I'm on 1.7.0 of PowerCalc and the following versions within HA:

Home Assistant: 2023.6.3 Supervisor: 2023.07.1 Operating System: 10.3 Frontend: 20230608.0 - latest

I'm wondering if to actually remove and re-add a few of the sensors just to see if that makes a difference. But as you say, these could be issues at a different level and not related to Powercalc. Will drop some feedback in the next couple of days.

Thanks for your reply.

bramstroker / homeassistant-powercalc