Since SENEC Application v0825,v0826 & v0827 no STATISTIC data is provided [No Data for multiple Sensors]

[marq24]

Problem

When your SENEC.Home will be updated to v0825 (also after the update to v0826) the integration is not able to provide data for a couple of sensors..

Affected Sensors (that's all the total sum data displayed in kWh):

• House consumed
• Solar generated
• Battery charged
• Battery discharged
• Grid Imported
• Grid Exported
• Wallbox charged

The reason behind is, that SENEC has decided to exclude 'STATISTIC' information from the response of the 'lala.cgi'... That's quite sad, since these sensors are mainly used for the Home Assistent Energy View...

Right now there is no quick fix to this situation - but since also data for 'DISPLAY' is missing my guess is, that SENEC is just building so called Banana-Software. Without "DISPLAY" information from the lala.cgi the /Display.html does not work at all - so I have hope, that the empty response parts have slipped through QA/QC

Current Solutions to access your statistics data

Please read the complete post/Issue text - since at the bottom you will find important information for you Energy Dashboard

'mein-senec.de' WebAPI

When you update this Integration (ha-senec-v3) to the latest version you have the option to add an additional integration entry. When you start the process you have to select 'WEB.API: mein-senec.de Portal (usable with all SENEC.Home variants)' from the selection list and then provide your mein-senec.de credentials. This will add 13 additional Sensors to your Home Assistant (currently wallbox data is missing - cause I don't have one (yet)) - when you know the fields in the web-api - just let me know. Thanks to @mstuettgen!

Mapping lala.cgi to web-api Sensor names

Please find below a table comparing the sensor names between the classic integration and the web-api based integration - this comparison assume, that you did not change the names ['senec' & 'senec_wabapi'] since they will be used as prefixes

Name	lala.cgi based	web-api
House consumed	sensor.senec_house_total_consumption	sensor.senec_wabapi_consumption_total
Solar generated	sensor.senec_solar_total_generated	sensor.senec_wabapi_powergenerated_total
Battery discharged	sensor.senec_battery_total_discharged	sensor.senec_wabapi_accuimport_total
Battery charged	sensor.senec_battery_total_charged	sensor.senec_wabapi_accuexport_total
Grid Imported	sensor.senec_grid_total_import	sensor.senec_wabapi_gridimport_total
Grid Exported	sensor.senec_grid_total_export	sensor.senec_wabapi_gridexport_total
Wallbox charged	sensor.senec_wallbox_energy	-not-implemented-yet-

Riemann Sum Integal Platform

AS WORKARROUND you can add the following section to your ha configuration.yaml (if you have already a sensor section obviously you just need to add the sensor's itself) - unfortunatly the the integration platform does not allow to provide icon's - so you have to edit them manually later via the GUI (if you need/want them)...

Please note, that the source sensor names based on the default integration name 'senec' - if you modified the name during the initial setup you need to adjust the source sensors.

sensor:
  - platform: integration
    source: sensor.senec_house_power
    name: senec_house_total_consumption_fb
    unique_id: senec_house_total_consumption_fb
    #icon: mdi:home-import-outline
    unit_prefix: k
    round: 2

  - platform: integration
    source: sensor.senec_solar_generated_power
    name: senec_solar_total_generated_fb
    unique_id: senec_solar_total_generated_fb
    #icon: mdi:solar-power
    unit_prefix: k
    round: 2

  - platform: integration
    source: sensor.senec_battery_charge_power
    name: senec_battery_total_charged_fb
    unique_id: senec_battery_total_charged_fb
    #icon: mdi:home-battery
    unit_prefix: k
    round: 2

  - platform: integration
    source: sensor.senec_battery_discharge_power
    name: senec_battery_total_discharged_fb
    unique_id: senec_battery_total_discharged_fb
    #icon: mdi:home-battery-outline
    unit_prefix: k
    round: 2

  - platform: integration
    source: sensor.senec_grid_imported_power
    name: senec_grid_total_import_fb
    unique_id: senec_grid_total_import_fb
    #icon: mdi:transmission-tower-import
    unit_prefix: k
    round: 2

  - platform: integration
    source: sensor.senec_grid_exported_power
    name: senec_grid_total_export_fb
    unique_id: senec_grid_total_export_fb
    #icon: mdi:transmission-tower-export
    unit_prefix: k
    round: 2

#  - platform: integration
#    source: sensor.senec_???
#    name: senec_wallbox_energy_fb
#    unique_id: senec_wallbox_energy_fb
#    icon: "mdi:ev-station"
#    unit_prefix: k
#    round: 2    

Handling the change of the EntityIDs in the Energy Dashboard

Since we have the situation, that some SENEC.Home Systems still return STATISIC data (since they are not updated) and that this data might come back with another Update - who knows? - the EntityIDs of the new Sensors (that have been added in both cases (WebAPI or Riemann)) are different by purpose.

When you want to keep your historic data in the Energy Dashboard (and I guess you want) then the current known best way is described here in an additional issue: How to configure Energie Dashboards (after SENEC.Home have been updated to 0825+)

[electricdelicate]

Oh no ....

[neubi4]

If i got is correctly, these are all "daily usage" statistics data, e.g. how many kWh House consumed today.

This should be replaceable by using an Home Assistant Riemann sum integral (https://www.home-assistant.io/integrations/integration/#energy) which simply can summarize House Power (W) to kWh like in the linked example.

Since this counts infinitely higher, to get an daily used you can use Utility Meter (https://www.home-assistant.io/integrations/utility_meter/) to get an sensor on the integral which is resetted daily/monthly value.

These Utility Meter sensors then could be used in Home Assistents Energy View.

We cloud provice YAML examples for this in the readme of this repo.

[neubi4]

After looking at the old data, it seems that the Utility Metern is not needes since the old data also only counts upwards infinitely.

So it should be fine only using the Riemann sum integral sensors. I will try this.

[marq24]

FUNNY - while I am tyring - you already provide the info...

the values (that are still available in my environment) are constantly growing - so there is no need to reset them on a monthly/weekly/daily/hourly basis - so looks like using the suggested integration from @neubi4 sounds like a decent replacement...

Once the http-v3 build is confirmed as "working" I can spend some time working on example yaml parts that you can copy&paste in your ha configuration.yaml...

[electricdelicate]

Sounds great, thank you for your support!

[marq24]

added workaround information in the initial post

[egonkab]

hello marq24,

thanks for your hard work. It works very well. I use Powercalc to calculate the total sensors, this also works well.

[sigcarunchio]

Hello marq24, just discovered your fork after firmware update, great work!

1) About missing entities, i also redo them using custom sum integration templates, but instead using yaml code, i use helpers; do you think it's the same?

2) My problem is that i have a master and a slave senec v3; i think that missing statistics gave already total values, now (but i confirm this later) with custom sensor i have only data about master senec (the one that i register in the integration); do i need to register also slave battery and do total custom sensors?

3) Worst thing when i change sensors in energy panel with custom ones, I LOST all past data of about one year!!!!! sigh!!!!

anyway again great work, thank you!

[neubi4]

Hi @sigcarunchio

1.: yes, using Helpers and the yaml code is the same.

3.: i left the old sensors with their current VARIABLE_NOT_FOUND value in the energy dashboard and only added the new helper sensors. This way i still have the old stats data in my energy dashboard. I dont know exactly how the dashboard works internally, it is possible that readding the old sensors will not work for you.

2.: This is an interesting one since not many users have multiple systems. Does the slave senec also has an seperate IP and the webinterface? If yes, then i would try to add the slave additionally to the master. To get the total stat values of both systems, you could try to calculate the sum of the values from both senecs and calculate the Riemann sum integral on this value.

[sigcarunchio]

2) Yes, it has a separate IP with the same web interface, try this way!

3) Ok, i replaced them, try to reinsert also old sensors

[sigcarunchio]

Readded also old sensors, past stats availables again! As soon as i will add also slave senec, I will report if it works

[electricdelicate]

added workaround information in the initial post

Good job, work also like a charm. One question, is the "Riemann sum" exact enough for a proper calculation? I will compare that to the Senec App.

[bobthebuilder4711]

I wrote a cautioning comment over on the old integration about Riemann sums: https://github.com/mchwalisz/home-assistant-senec/issues/47#issuecomment-1701014858

So here just a short summation: Riemann sums will start to differ from what the battery reports with time and a better solution would be to simply get that data from mein-senec.de.

mein-senec.de only updates every 5 minutes, but it's only for the totals, so that should be fine. The other values can still be queried from the lala.cgi if possible.

[bobthebuilder4711]

It also seems like the update reset the settings for our Home V2.1 to poll our three SMA inverters. Solar Power just stays at 0W. Anyone else has this problem?

Don't have the installer's password for the senec battery, so I guess I'll have to add the inverters to home assistant directly...

Edit: Okay, Solar Power is back. Seems like the battery needed an hour to work properly again.

[Kuechenfenster]

https://github.com/mchwalisz/home-assistant-senec/issues/47#issuecomment-1701014858

[marq24]

Please note, there was a typo in my provided configuration.yaml

in the senec_grid_total_export_fb the source has to be: source: sensor.senec_grid_exported_power (and not source: sensor.senec_senec_grid_exported_power) - sorry...

[marq24]

Since here (locally) I still have the http version - I am able to compare the values [created via the integration platform vs data from the senec statistics json data]

[Kuechenfenster]

@Kuechenfenster

mchwalisz#47 (comment)

could you let us know, what you wanted to say?

my apology.. please ignore it.

[electricdelicate]

added workaround information in the initial post

Good job, work also like a charm. One question, is the "Riemann sum" exact enough for a proper calculation? I will compare that to the Senec App.

As far as I can tell at my system, the Riemann sum is only correct at my charging value, the discharging value is very deviating to MySenec/official App ....

[marq24]

here is a comparison of the 'discharging value' here @ my home... YES the numbers are different...

Just as Update (working on a better replacement) - I have spend already some hours trying to access mein-senec.de - but failed with the aiohttp python classes to get an appropriate login (with the request-lib this works smooth like silk) - one of the many redirects is failing

[egonkab]

Hello, the powercalc result from 09/01/2023

[marq24]

Passt über den dicken Daumen - oder? [It just took me a while to realize, that top part (of the image) is from SENEC-App and the lower part is from HA-Energy-Tab (I assume feeded by the 'integration' sensors...)

[egonkab]

Ich schwenke mal auf deutsch um. Die Werte aus dem HA Energy Tab sind die der PowerCalc Berechnung. Ich habe am zum Vergleich folgende Sensoren aufgesetzt (senec_grid_exported_power), Ergebnis:

PowerCalc: 14,53kWh - RMS Trapez: 14,12kWh - RMS Left: 14,19kWh - Senec App: 14,5kWh

PowerCalc ist am nächsten dran, warum kann ich aber nicht sagen. Abfrage Intervall vom Speicher sind 2 Sekunden.

[BMeyendriesch]

Könntest Du mal bitte die powercalc.yaml für obige Ausgabe mitteilen.

[BMeyendriesch]

Hat sich erledigt. Ich hatte mir mit gleichzeitigen Konfigurationsversuchen per GUI und yaml selbst auf dem Fuß gestanden . . .

[marq24]

hier teilen wäre 'trotzdem' schön ;-) - Gute Morgen btw..

[marq24]

updated initial post -> added a solution alternative: mein-senec.de WebPortal Integration Entry

[BMeyendriesch]

Ich habe hier einen Haufen von Helpern und darauf aufbauenden Sankey-Diagrammen, die auf den ursprünglichen Entities der v2.1-Integration

sensor.senec_solar_total_generated
sensor.senec_house_total_consumption
sensor.senec_grid_total_import
sensor.senec_grid_total_export
sensor.senec_battery_total_charged
sensor.senec_battery_total_discharged

basieren und würde nur sehr ungern sämtliche Abhängigkeiten von diesen Entities anfassen und modifizieren. Gibt es nicht eine schlanke Methode, die Ausgangswerte von powercalc in diese Entities umzuleiten? Bisher habe ich powercalc nicht dazu überreden können. Möglicherweise habe ich auch das gesamte Konzept der Entities in Homeassistant nicht hinreichend verstanden . . .

[marq24]

basieren und würde nur sehr ungern sämtliche Abhängigkeiten von diesen Entities anfassen und modifizieren. Gibt es nicht eine schlanke Methode, die Ausgangswerte von powercalc in diese Entities umzuleiten? Bisher habe ich powercalc nicht dazu überreden können. Möglicherweise habe ich auch das gesamte Konzept der Entities in Homeassistant nicht hinreichend verstanden . . .

ich verstehe Deine Herausvorderung - mit der gerade veröffentlichten web-api variante kommst Du hier 'so' erstmal nicht weiter, da die entitie_id's der web-api sensoren 'natürlich' nicht den namen der 'verlorengegangenen' entsprechen...

Da ich zu Hause noch die "alte" (nicht https) variante habe, kann ich auch nicht einfach diese bestehenden Sensoren überschreiben... Ich muss mal einwenig überlegen, ob es da noch eine elegante/end-user-freundliche variante gibt. Aber dazu benötige ich etwas Zeit.

[electricdelicate]

perfekt, ich habe ein zusätzliches WebAPI Device hinzugefügt.

[bobthebuilder4711]

Ich habe hier einen Haufen von Helpern und darauf aufbauenden Sankey-Diagrammen, die auf den ursprünglichen Entities der v2.1-Integration

sensor.senec_solar_total_generated
sensor.senec_house_total_consumption
sensor.senec_grid_total_import
sensor.senec_grid_total_export
sensor.senec_battery_total_charged
sensor.senec_battery_total_discharged

basieren und würde nur sehr ungern sämtliche Abhängigkeiten von diesen Entities anfassen und modifizieren. Gibt es nicht eine schlanke Methode, die Ausgangswerte von powercalc in diese Entities umzuleiten? Bisher habe ich powercalc nicht dazu überreden können. Möglicherweise habe ich auch das gesamte Konzept der Entities in Homeassistant nicht hinreichend verstanden . . .

Bau Dir Template Sensoren, deren IDs Du dann in deine Helper einbauen kannst. Dann kannst Du relativ einfach in den Template Sensoren ändern, wenn sich wieder was ändert, ohne dass Du die Helper nochmal anfassen musst.

zum Beispiel (in der configuration.yaml eintragen):

template:
  - sensor:
    - name: "Fester Sensor PV Erzeugung Gesamt"
      unique_id: "hier was Einzigartiges ausdenken"
      unit_of_measurement: kWh
      device_class: energy
      state_class: total_increasing
      state: >
        {{ float(states('sensor.senec_solar_total_generated')) }}

In der letzten Zeile kannst Du dann den jeweiligen Sensor eintragen, den Du eigentlich nutzen willst.

Home Assistant baut die entity ID aus dem name zusammen. In dem Fall wäre das dann sensor.fester_sensor_pv_erzeugung_gesamt. [edit (marq24) - wenn Du unter 'unique_id' nichts angibst - sonst ist die Entity ID natürlich das was Du in dem entsprechenden Attribut hinterlegt hat Bob weiss sehr genau was er hier schreibt - und ich nicht!][edit (bob): Die 'unique_id' und die 'entity_id' sind zwei verschiedene paar Schuhe. Die unique_id braucht Home Assistant intern zur eindeutigen Identifizierung, selbst wenn man eine 'entity_id' angibt. Meine Templates zum Beispiel haben da UUIDs drin stehen (zB: "49da82ca-4d3b-4db3-ac86-e8c3b0dafd50"). Gibt es keine 'unique_id', dann kann es sehr gut sein, dass Home Assistant meckert, dass man den sensor für viele Dinge nicht benutzen kann.]

Edit: Aber definitiv vorher mal ausprobieren mit einem Template Sensor. Vor allem bei der unit_of_measurement weiß ich gerade nicht, ob Du kWh oder nur Wh rein bekommst.

[marq24]

[edit (bob): Die 'unique_id' und die 'entity_id' sind zwei verschiedene paar Schuhe. Die unique_id braucht Home Assistant intern zur eindeutigen Identifizierung, selbst wenn man eine 'entity_id' angibt. Meine Templates zum Beispiel haben da UUIDs drin stehen (zB: "49da82ca-4d3b-4db3-ac86-e8c3b0dafd50"). Gibt es keine 'unique_id', dann kann es sehr gut sein, dass Home Assistant meckert, dass man den sensor für viele Dinge nicht benutzen kann.]

wieder was über HA gelernt! - danke Bob!

[bobthebuilder4711]

Hatte halt selber schon das Vergnügen, stundenlang dumm auf den Bildschirm zu starren, als ich meine ESPEasy Steckdosen per MQTT an HA anbinden wollte und dann in der GUI nicht richtig benutzen konnte. Da stößt man dann schnell auf diese Seite, die es gut erklärt: https://www.home-assistant.io/faq/unique_id/

Edit:

Wegen den Wallboxen: Wir haben im MQTT Script die Links herausgefunden, indem wir einfach auf mein-senec.de Rechtsclick->Inspizieren gemacht haben und dann auf dem Netzwerk Tab geschaut haben, welche URLs gepollt werden. So hätte man das mit den Wallboxen auch machen können.

Aber hier habe ich die URLs gefunden, die man aufrufen muss inklusive der URLs für die Wallbox(en): https://documenter.getpostman.com/view/10329335/UVCB9ihY

[blackfed]

Hi,

Just started using today this integration, I have a V2 running firmware 825, so I set it up using the WebAPI option by logging in. As soon as it creates the device in my HASS I can see data on the sensors, but then it doesn't get any other update, and after a couple of minutes they all go "not available". Is there something I'm doing wrong?

Thanks

[egonkab]

So wieder ein bisschen Zeit für den JA

updated initial post -> added a solution alternative: mein-senec.de WebPortal Integration Entry

Wie hoch ist die Abfragerate?

[marq24]

Wie hoch ist die Abfragerate?

5 minutes

[bobthebuilder4711]

For my V2.1 the portal updates ever 5 minutes. But I've been told that you can poll the portal much quicker if you have a V4. 30 seconds seems to still be okay. But I can't say if the portal updates even quicker than that.

@egonkab you should definitely use both the webportal for the kWh data and the normal local connection for the kW data since that updates much faster. I'm currently polling the local connection every 10 seconds without any problems.

[jc2308]

Hi, today i ran into this issue. So far i got the things to work now, but how can i add the WepApi? i can´t chose it by adding a new integration. There are only the 3 Senec variants.

My integration verision is 3.0.5 and beta is active.

[marq24]

well the screenshot is from 3.0.5 - so what ever you did (when you tried to install 3.0.6-test-xx) did not work - might have forgotten to restart HA? Tried to use SHIFT+Reload in the Browser?

[jc2308]

Oh, sry. i didn´t see that there is a new Version. The latest version in HACS is 3.0.5 for me. I´ll have a look at the new version. Thanks for your Work :)

[marq24]

Oh, sry. i didn´t see that there is a new Version. The latest version in HACS is 3.0.5 for me. I´ll have a look at the new version.

the "new build" is currently just in BETA-state... so that's probably the reason why you did not see/saw it...

[sigcarunchio]

Hi marq24, i'm trying the new webapi sensors, but value doesn't match the ones of senec ios app. They are lower than app; i have two senec modules, a master and a slave; with the old integration missing values reported total values of produced energy, battery in/out, grid in/out; actual webapi value are total?

Also, is this correct?

Battery discharged --> sensor.senec_battery_total_discharged --> sensor.senec_wabapi_accuimport_total Battery charged --> sensor.senec_battery_total_charged --> sensor.senec_wabapi_accuexport_total

or new sensors are inverted? Thank you!

[marq24]

Hi marq24, i'm trying the new webapi sensors, but value doesn't match the ones of senec ios app. They are lower than app; i have two senec modules, a master and a slave; with the old integration missing values reported total values of produced energy, battery in/out, grid in/out; actual webapi value are total?

please see the conversation #8 concerning the progress/state of the support for "multiple" stacked SENEC Systems...

Also, is this correct?

Battery discharged --> sensor.senec_battery_total_discharged --> sensor.senec_wabapi_accuimport_total Battery charged --> sensor.senec_battery_total_charged --> sensor.senec_wabapi_accuexport_total

or new sensors are inverted?

this question should go to SENEC master-brain developed the (internal)API... I fully agree with you, that (internal) naming is quite confusing... All I can say (based on my personal observation) is, that the accuimport fields from the webapi containing data when my home is consuming power from the battery - and the accuexport fields when the battery will be charged...

So IMHO the SENEC developer took the perspective from the point of view of Main unit (and not from the Battery)...

[bobthebuilder4711]

Yes, the sensors are inverted to what we're used to.

The names are like that because mein-senec.de calls them that internally. And they correspond to how import and export works from the view of the house.

Think about it this way:

Energy flowing from your home into the grid is: gridexport.

So energy flowing from your home into the battery is: batteryexport.

And if you need energy from the grid then that is: gridimport.

If you need energy from the battery then that is: batteryimport.

[mstuettgen]

For my V2.1 the portal updates ever 5 minutes. But I've been told that you can poll the portal much quicker if you have a V4. 30 seconds seems to still be okay. But I can't say if the portal updates even quicker than that.

I can confirm that you can poll data from mein-senec.de at least every 30 seconds if you have a V4. But I also have not tried shorter polling intervals, as 30s is absolutely fine for me. Maybe the V4 transmits its data more often "back home"? I might investigate that when I return from my vacation.

If you wanna play around with it yourself, run "senec_webgrabber.py" from https://github.com/mstuettgen/homeassistant-addons/blob/main/senecweb2mqtt/senec_webgrabber.py. You just need to fill in your senec credentials.

The script will print out the that data it could retrieve from the website to your console.

[marq24]

For my V2.1 the portal updates ever 5 minutes. But I've been told that you can poll the portal much quicker if you have a V4. 30 seconds seems to still be okay. But I can't say if the portal updates even quicker than that.

I can confirm that you can poll data from mein-senec.de at least every 30 seconds if you have a V4. But I also have not tried shorter polling intervals, as 30s is absolutely fine for me. Maybe the V4 transmits its data more often "back home"? I might investigate that when I return from my vacation.

thanks! - I actually have already (this morning) adjusted the polling interval to 60sec if the system identifies itself as 'Home V4'... but this only takes effect, when you re-install the integration entry [since the interval is currently not part of the configuration]

[Thomk-63]

HI, I am now using the WebAPI for getting the statistics values. This seems to work and I can take these WebAPI sensors for the energy dashboard. BUT: All the energy dashboard history is then gone, as this is saved under the original senec entity IDs. Is there any way to write the WebAPI values to the senec entity IDs. Ideally this would be done directly in the integration by mapping the WebAPI to the unavailable statistics entity IDs. Unfortunately, I am not a software programmer. Any idea?

[bobthebuilder4711]

Seems like you can just delete the old sensors and then rename the new sensors to have the entity_ids of the old ones: https://community.home-assistant.io/t/migrate-lts-history-to-replacement-sensor/460115/3

And if one sensor is in kWh and the other is in Wh then use a template sensor for that as I described earlier in German or here in English: https://github.com/mchwalisz/home-assistant-senec/issues/47#issuecomment-1709703085

But definitely create a backup before you try any of this.

Edit:

I just tried this with the house consumption sensor because that is calculated by home assistant anyway and thus not important for me if it broke. If you rename the new sensor's entity_id to the entity_id of the old sensor then you do in fact have the old historical data on that new sensor. But: If you've already gathered data points in the last few days then those aren't transferred over.

All the energy dashboard history is then gone, as this is saved under the original senec entity IDs.

To combat that just leave the old entities in the energy dashboard. I have both the old and the new sensors in there and it looks like this on the day of the update. It's yellow instead of orange from there on but that doesn't really bother me.

In the future someone with too much time on their hands might write some SQL code to migrate the long term data from the old entity to the new one. But I don't see me doing that in the next few weeks or even months.

[marq24]

> To combat that just leave the old entities in the energy dashboard. I have both the old and the new sensors in there and it looks like this on the day of the update. It's yellow instead of orange from there on but that doesn't really bother me. !

exactly! - as documented here - https://github.com/marq24/ha-senec-v3/issues/11 - thanks @bobthebuilder4711 for your help here supporting the integration!

Once more I have edited my initial text of this issue in order to also inform you about the Energy Dashboard...

[SteffenS27]

Hi, sorry for confusing that technical discussion. Bit can someone please tell what I have to do with that beta version and so on ? Sorry bit HA is new for me