Error on wnsm/live_sensor.py:68

[akicker]

Logger: custom_components.wnsm.live_sensor Quelle: custom_components/wnsm/live_sensor.py:68 Integration: WienerNetzeSmartmeter (Dokumentation, Probleme) Erstmals aufgetreten: 9. April 2024 um 21:46:21 (7 Vorkommnisse) Zuletzt protokolliert: 09:46:20

Please file an issue with this error and (anonymized) payload in github {'hasSmartMeter': True, 'isDataDeleted': False, 'zaehlpunkt': 'AT001000000000000000xxxxxxxxxxxxxx', 'type': 'TAGSTROM', 'address': 'Adresse', 'zip': '1130'} {'consumptionYesterdayValue': 585, 'consumptionYesterdayValidated': True, 'consumptionYesterdayTimestamp': '2024-04-07T22:00:00.000Z', 'consumptionDayBeforeYesterdayValue': 495, 'consumptionDayBeforeYesterdayValidated': True, 'consumptionDayBeforeYesterdayTimestamp': '2024-04-06T22:00:00.000Z'} {'lastValue': 4019380, 'lastReading': '2024-04-08T22:00:00.000Z', 'lastValidated': True, 'lastType': '1-2:1.8.0'} {'optIn': True, 'consumptionAverage': 0, 'consumptionMinimum': 0, 'consumptionMaximum': 0} Please file an issue with this error and (anonymized) payload in github {'hasSmartMeter': True, 'isDataDeleted': False, 'zaehlpunkt': 'AT001000000000000000xxxxxxxxxxxxxx', 'type': 'TAGSTROM', 'address': 'Adresse', 'zip': '1130'} {'consumptionDayBeforeYesterdayValue': 585, 'consumptionDayBeforeYesterdayValidated': True, 'consumptionDayBeforeYesterdayTimestamp': '2024-04-07T22:00:00.000Z'} {'lastValue': 4019380, 'lastReading': '2024-04-08T22:00:00.000Z', 'lastValidated': True, 'lastType': '1-2:1.8.0'} {'optIn': True, 'consumptionAverage': 0, 'consumptionMinimum': 0, 'consumptionMaximum': 0} Please file an issue with this error and (anonymized) payload in github {'hasSmartMeter': True, 'isDataDeleted': False, 'zaehlpunkt': 'AT001000000000000000xxxxxxxxxxxxxx', 'type': 'TAGSTROM', 'address': 'Adresse', 'zip': '1130'} {'consumptionYesterdayValue': 505, 'consumptionYesterdayValidated': True, 'consumptionYesterdayTimestamp': '2024-04-08T22:00:00.000Z', 'consumptionDayBeforeYesterdayValue': 585, 'consumptionDayBeforeYesterdayValidated': True, 'consumptionDayBeforeYesterdayTimestamp': '2024-04-07T22:00:00.000Z'} {'lastValue': 4019885, 'lastReading': '2024-04-09T22:00:00.000Z', 'lastValidated': True, 'lastType': '1-2:1.8.0'} {'optIn': True, 'consumptionAverage': 0, 'consumptionMinimum': 0, 'consumptionMaximum': 0}

[MatthiasRei]

Today the smartmeter Webportal from Wienernetze is offline. It looks like something is going on there :)

[GeraldPape]

I have this error today,.... 2024-04-12 08:26:01.016 ERROR (MainThread) [custom_components.wnsm.live_sensor] Unable to load consumption 2024-04-12 08:26:01.017 ERROR (MainThread) [custom_components.wnsm.live_sensor] Please file an issue with this error and (anonymized) payload in github {'hasSmartMeter': True, 'isDataDeleted': False, 'name': 'Bezug', 'zaehlpunkt': 'AT00100000000000000010000xxxxxxx6', 'type': 'TAGSTROM', 'address': 'Straße 45, 1220 Wien', 'zip': '1220'} {'consumptionYesterdayValue': 705, 'consumptionYesterdayValidated': True, 'consumptionYesterdayTimestamp': '2024-04-10T22:00:00.000Z', 'consumptionDayBeforeYesterdayValue': 1563, 'consumptionDayBeforeYesterdayValidated': True, 'consumptionDayBeforeYesterdayTimestamp': '2024-04-09T22:00:00.000Z'} {'lastValue': 11138193, 'lastReading': '2024-04-11T22:00:00.000Z', 'lastValidated': True, 'lastType': '1-2:1.8.0'} {'optIn': True, 'consumptionAverage': 0, 'consumptionMinimum': 0, 'consumptionMaximum': 0}

[reox]

Maybe something changed in the API or it has temporary issues... I get more errors, but in general the import still works.

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 951, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1268, in async_device_update
    await self.async_update()
  File "/config/custom_components/wnsm/live_sensor.py", line 29, in async_update
    zaehlpunkt = await self.get_zaehlpunkt(smartmeter)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/base_sensor.py", line 108, in get_zaehlpunkt
    zaehlpunkte = self.contracts2zaehlpunkte(contracts)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/base_sensor.py", line 93, in contracts2zaehlpunkte
    geschaeftspartner = contracts[0]["geschaeftspartner"] if "geschaeftspartner" in contracts[0] else None
                                                                                    ~~~~~~~~~^^^
KeyError: 0

Traceback (most recent call last):
  File "/config/custom_components/wnsm/statistics_sensor.py", line 113, in async_update
    zaehlpunkt = await self.get_zaehlpunkt(smartmeter)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/base_sensor.py", line 108, in get_zaehlpunkt
    zaehlpunkte = self.contracts2zaehlpunkte(contracts)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/base_sensor.py", line 92, in contracts2zaehlpunkte
    raise RuntimeError(f"Cannot access Zaehlpunkt {self.zaehlpunkt}")
RuntimeError: Cannot access Zaehlpunkt AT[...]

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 951, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1268, in async_device_update
    await self.async_update()
  File "/config/custom_components/wnsm/live_sensor.py", line 28, in async_update
    await self.hass.async_add_executor_job(smartmeter.login)
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/api/client.py", line 125, in login
    code = self.credentials_login(url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/api/client.py", line 75, in credentials_login
    raise SmartmeterLoginError("Login failed. Check username/password.")
custom_components.wnsm.api.errors.SmartmeterLoginError: Login failed. Check username/password.

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 951, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1268, in async_device_update
    await self.async_update()
  File "/config/custom_components/wnsm/live_sensor.py", line 28, in async_update
    await self.hass.async_add_executor_job(smartmeter.login)
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/api/client.py", line 124, in login
    url = self.load_login_page()
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/api/client.py", line 53, in load_login_page
    action = tree.xpath("(//form/@action)")[0]
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

[DarwinsBuddy]

Is this still a problem? I checked my set up and couldn't find any concerning logs. Something's definitely off with the API. I just want to re-confirm if it was a temporary fluke or there's something we can do from our side (if it still persists). @akicker @MatthiasRei @GeraldPape @reox

[MatthiasRei]

No problems now 👍

[reox]

I guess that the API has some downtimes or sometimes returns not all data... The errors I collected above are probably things which could be programmed in a more resilient way, for example by checking returned data. But as I said, the import still works for me!

[GeraldPape]

Immer noch: 2024-04-15 14:37:10.344 ERROR (MainThread) [custom_components.wnsm.live_sensor] Unable to load consumption 2024-04-15 14:37:10.345 ERROR (MainThread) [custom_components.wnsm.live_sensor] Please file an issue with this error and (anonymized) payloa

aber ... ich habe zwei Smartmeter in meinem Vertrag. Dieser Fehler kommt beim Verbrauch. Einspeisen bekomme ich gar keine Daten.

[akicker]

ich habe einen Smartmeter, allerdings 2 Zählpunkte - Verbrauch und Einspeisung Daten für den Verbrauch werden geladen, Einspeisung nicht! PS: im Webgui (https://smartmeter-web.wienernetze.at/) sind beide Daten vorhanden.

[W-M-B]

ich habe einen Smartmeter, allerdings 2 Zählpunkte - Verbrauch und Einspeisung Daten für den Verbrauch werden geladen, Einspeisung nicht! PS: im Webgui (https://smartmeter-web.wienernetze.at/) sind beide Daten vorhanden.

@akicker if the problem is not related to topic of the issue, please create a new one and provide information that helps to understand the problem. or this is maybe related to Hallo, wenn der Fehler mit den 2 Smartmeter nicht mit dem Titel zusammenhängt, bitte einen neuen Issue thread mit ausreichend Details beginnen. Eventuell hat es auch mit Issue 191 zu tun?

https://github.com/DarwinsBuddy/WienerNetzeSmartmeter/issues/191

[akicker]

Mehrere Warnungen und Fehler nach dem Start: Starting import of historical data. This might take some time. wnsm: Error on device update! Setup of sensor platform wnsm is taking over 10 seconds. FEHLER Unable to load consumption FEHLER Please file an issue with this error and (anonymized) payload in github ...

[akicker]

New WARNING since core 2024.5.1: HomeAssistantType was used from wnsm, this is a deprecated alias which will be removed in HA Core 2025.5. Use homeassistant.core.HomeAssistant instead, please report it to the author of the 'wnsm' custom integration

[tschoerk]

Also getting this Error.

Logger: custom_components.wnsm.live_sensor Quelle: custom_components/wnsm/live_sensor.py:68 Integration: WienerNetzeSmartmeter

2024-05-07 10:38:08.917 ERROR (MainThread) [custom_components.wnsm.live_sensor] Unable to load consumption
2024-05-07 10:38:08.918 ERROR (MainThread) [custom_components.wnsm.live_sensor] Please file an issue with this error and (anonymized) payload in github 
{'hasSmartMeter': True, 'isDataDeleted': False, 'zaehlpunkt': 'AT001000000000000000#############', 'type': 'TAGSTROM', 'address': 'XXXXXXXXXXXXXXXXXX', 'zip': '####'} 
{'consumptionYesterdayValue': 2588, 'consumptionYesterdayValidated': True, 'consumptionYesterdayTimestamp': '2024-05-05T22:00:00.000Z', 'consumptionDayBeforeYesterdayValue': 2445, 'consumptionDayBeforeYesterdayValidated': True, 'consumptionDayBeforeYesterdayTimestamp': '2024-05-04T22:00:00.000Z'} 
{'lastValue': 6495082, 'lastReading': '2024-05-06T22:00:00.000Z', 'lastValidated': True, 'lastType': '1-2:1.8.0'} {'optIn': True, 'consumptionAverage': 0, 'consumptionMinimum': 0, 'consumptionMaximum': 0}

[tschoerk]

Ok so figured it out after some tinkering. This problem only happens when there are two Zählpunkte. They way that the live sensor update works is that it gets the basic data from Wiener Netze, which includes the last value of the meter readings for the "main" or "favourite" Zählerpunkt. If the Zählerpunkt from the sensor and the "main" Zählerpunkt are identical then this value is used to update the sensor.

If the second Zählerpunkt is updated, the base meter readings don't have the information for this Zählerpunkt, since it's not the "main" Zählerpunkt. So it tries to get the information by checking the consumption data, which shows the consumption starting from the chosen day, in this case for yesterday, to update the sensor with this value. This second method has several bugs though:

1. live_sensor.py checks if the return of get_consumption_raw has a field statistics if ("values" in verbrauch_raw and "statistics" in verbrauch_raw). The raw response has this field, but the method get_consumption_raw maps this response with ATTRS_VERBRAUCH_CALL, which removes the statistics, so it never gets past the if. Therefore the return of get_consumption_raw should be unmapped and simply return the response. So change the return of the method get_consumption_raw in base_sensor.py from return translate_dict(response, ATTRS_VERBRAUCH_CALL) to return response.
2. WienerNetze returns the consumption from yesterday only when the start date is set to the day before yesterday, maybe because the API excludes the given start day and returns data starting from the day after. Maybe it's depending on the time of day, but I tested several times and it never returns data, when the start day is yesterday. So the start date needs to be set two days before today, to get yesterday's consumption. So changing verbrauch_raw = await self.get_consumption_raw(smartmeter, before(today())) to verbrauch_raw = await self.get_consumption_raw(smartmeter, before(before(today()))) in live_sensor.py should fix this. This one is still a bit buggy though, since the update from wiener netze doesn't happen until several hours into the day (5 AM last night for example) and every call after midnight returns no data until the data is available and therefore throws hourly errors.
3. Furthermore the safeget call to receive the average from the API response threw errors . Since safeget expects *args, there's no need to call this method with a list. Therefore the call should be changed from avg = safeget(verbrauch_raw, [["statistics"], ["average"]]) to avg = safeget(verbrauch_raw, "statistics", "average")
4. Since the consumption is in Wh but the default unit for the sensors in HA is kWh the final state must be divided by 1000: self._state = yesterdays_sum / 1000

With all those fixes the daily live updates work with the second Zählpunkt.

The inherent problem though is that the first and second Zählerpunkt give out completly different values, meaning the first one updates using the meter readings, so using the full consumption value since implementation of the meter and the second method only returns yesterdays consumption which are two different value types for the same sensor type.

So I had the idea to completely change how getting the value of the second Zählpunkt works. With a simple PUT call the second Zählerpunkt can be set as the "main" Zählerpunkt. Afterwards the calls for base_information and meter_readings return the full information regarding this one. Afterwards the first Zählerpunkt can be set as "main" again. That would give both sensors the same values.

Here is the call:

PUT https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/kundennummer/{customer_id}/zaehlpunkt/default
Body: {"zaehlpunktNummer":"AT001000000000000000#############"}

So instead of making a pull request for the fixes stated above (you can find the fixes stated above in my repo https://github.com/tschoerk/WienerNetzeSmartmeter/tree/bugfix/loadConsumption if interested), I would work on the alternative method, since I think it makes much more sense that way. I will make a request when finished.

[tschoerk]

I tested different solutions with my proposed idea of changing the main or default zaehlerpunkt to get the meter readings, but I can't figure out how to do it properly in the async environment. There's a race condition when changing the default zaehlpunkt so both sensors get the same data. There is some wonky solution using asyncio.sleep(), but it's rather buggy. Maybe someone else has some ideas regarding that?

Anyway I will make a pull request for my proposed fixed above. Both values still provide the daily usage in different forms, so it's probably fine.