Energy recordings (consumption) enabled for ems-esp gateways

[tp1de]

With adapter version >= 2.3.1 energy consumption processing and statistics for the ems-esp gateways are enabled.

Similar to the "recordings" within the original Bosch gateways the energy statistics are processed in a similar way. Recordings take samples of modulation / boiler power every minute, for ems-esp this samples are taken now every 15 seconds. The result is within objects substructure "energy".

As a result, the power consumption is even more accurate than recordings. I will go on testing and comparing.

The recordings provide historic consumptions, the ems-esp statistics are collected when you start this function within the ioBroker adapter. Whenever the ems-esp gateway is down or the adapter is not running, statistics can't be collected.

Please report your experience with this new function.

[tp1de]

A Flot graph for hourly consumption:

http://iobroker:8082/flot/index.html?relativeEnd=now&range=4320&aggregateType=step&height=500&noBorder=noborder&hoverDetail=true&timeFormat=%25H&_vis_refts=1470226228812&l%5B0%5D%5Bid%5D=ems-esp.0.energy.actualPower._Hours&l%5B0%5D%5Boffset%5D=0&l%5B0%5D%5Bcolor%5D=%23d78878&l%5B0%5D%5Bthickness%5D=3&l%5B0%5D%5Bshadowsize%5D=3&l%5B0%5D%5Bname%5D=Verbrauch+Heizung&l%5B0%5D%5Byaxe%5D=left&l%5B0%5D%5BchartType%5D=bar&l%5B0%5D%5Baggregate%5D=max&l%5B0%5D%5BignoreNull%5D=0&l%5B0%5D%5BafterComma%5D=0&l%5B0%5D%5Bsmoothing%5D=0&l%5B0%5D%5Bfill%5D=0.6&l%5B0%5D%5Bunit%5D=kWh&l%5B0%5D%5Bxaxe%5D=bottom&l%5B0%5D%5Bpoints%5D=false&l%5B0%5D%5Bdashes%5D=false&l%5B0%5D%5BdashLength%5D=10&l%5B0%5D%5BspaceLength%5D=10&l%5B0%5D%5Bmin%5D=0&l%5B0%5D%5Bmax%5D=22&l%5B0%5D%5Binstance%5D=sql.0&l%5B0%5D%5Bmultiplicator%5D=1&l%5B1%5D%5Bid%5D=ems-esp.0.energy.actualDHWPower._Hours&l%5B1%5D%5Boffset%5D=0&l%5B1%5D%5Baggregate%5D=max&l%5B1%5D%5Bcolor%5D=%2349498d&l%5B1%5D%5Bmin%5D=0&l%5B1%5D%5Bthickness%5D=3&l%5B1%5D%5Bshadowsize%5D=3&l%5B1%5D%5Bunit%5D=kWh&l%5B1%5D%5BchartType%5D=bar&l%5B1%5D%5Byaxe%5D=off&l%5B1%5D%5Bxaxe%5D=off&l%5B1%5D%5BafterComma%5D=0&l%5B1%5D%5Bdashes%5D=false&l%5B1%5D%5BdashLength%5D=10&l%5B1%5D%5BspaceLength%5D=10&l%5B1%5D%5Bpoints%5D=false&l%5B1%5D%5Bname%5D=Verbrauch+Warmwasser&l%5B1%5D%5Bmax%5D=22&l%5B1%5D%5Bfill%5D=0.6&l%5B1%5D%5Binstance%5D=sql.0&l%5B1%5D%5Bmultiplicator%5D=1&l%5B1%5D%5BignoreNull%5D=&timeType=relative&useComma=false&zoom=false&noedit=false&animation=0&_refts%5B%5D=1510325377256&_refts%5B%5D=1623410894102&barWidth=0.2&aggregateSpan=3600&legColumns=4&x_labels_color=%23070707&y_labels_color=%23080808&legBgOpacity=0.5&live=60&title=Energieverbrauch+%2F+Stunde&titlePos=top%3A35%3Bleft%3A50&legend=nw

[Curufinwem]

Hi! Thanks for the great new feature. Unfortunately, I seem to have a small problem with duplicated values. It could well be I set up something wrong. Two examples from yesterday. This is my burner power from the last 24h:

There is one activity on it between 17:00 and 18:00 as well as one between 20:00 and 21:00.

The JSON on actualPower Hours object on iobroker looks fine: { "date": "2023-10-08 23 hrs", "val": 0 }, { "date": "2023-10-08 22 hrs", "val": 0 }, { "date": "2023-10-08 21 hrs", "val": 0 }, { "date": "2023-10-08 20 hrs", "val": 1.11 }, { "date": "2023-10-08 19 hrs", "val": 0 }, { "date": "2023-10-08 18 hrs", "val": 0 }, { "date": "2023-10-08 17 hrs", "val": 4.15 }, { "date": "2023-10-08 16 hrs", "val": 0 },

However, when I look into the history on the _Hours object I get this:

This is the results that land in the influxdb. I think I also saw it on the history adapter but I messed this up a bit so I will need to wait a bit for a new set there.

Settings are as following:

I tried to set the value in the red box to "Array von Werten" but it keeps switching back.

This was run on version 2.4.0.

[tp1de]

I can't test influxdb since I haven't installed. But please use the latest version to test (2.4.1). The energy statistics are processed in 2 steps.

1. every minute actual modulation is multiplied with nominal power and result is stored here. Please look with object browser:

2. Every hour this state is analysed with the ioBroker build-in function getHistory. This should work with history, sql and influxdb adapter. But I can test history and sql only.

Please check again with latest version and check the power state within object browser. You should get only one record per hour within _Hours always with timestamp xx:30. Could you delete the hourly values and restart the adapter? The result you posted looks weird. Which version of influxdb are you using?

[Curufinwem]

So with the old version power looks pretty good I would say:

• I have now updated to the latest version 2.4.1.
• Deleted the hours JSON value (should this be all gone or an empty array?)
• influxdb is version 2.7.1, influxdb adapter is 3.2.0

The burner will only turn on again around 18:00 so I will only update with changed settings in the evening. I might set up a container with an SQL database to doublecheck what is reported there (not sure I will manage today).

[tp1de]

If energy power is correct then the hourly values should be fine as well. I have installed influxdb 2.7.1 as well on my proxmox server and I will test now using influxdb as well. No need for you to install SQL database.

If possible please delete the _Hours values in influxdb. There are newly calculated every hour. Do the _Days values in influxdb contain the duplicated values from _Hours?

[Curufinwem]

Thanks a lot for your efforts!

I checked days and it looks strange as well:

These are all the hourly value != 0 from today and yesterday:

2023-10-08T05:30:00.000Z;1.36;true;
2023-10-08T07:00:00.000Z;1.36;true;
2023-10-08T15:30:00.000Z;4.15;true;
2023-10-08T17:00:00.000Z;4.15;true;
2023-10-08T18:30:00.000Z;1.11;true;
2023-10-08T20:00:00.000Z;1.11;true;
2023-10-09T04:30:00.000Z;1.57;true;
2023-10-09T05:30:00.000Z;0.44;true;
2023-10-09T06:00:00.000Z;1.57;true;
2023-10-09T06:30:00.000Z;0.89;true;
2023-10-09T08:00:00.000Z;0.89;true;

6.62 in the days would be all values from the 8th if only counted once. But why at 14:00 and why are there two per day?

I will now delete the records from influxdb.

[Curufinwem]

So after deleting the records they have been recreated and now look better:

I will wait a bit because I am pretty sure that the duplicate values usually appear with a delay of about 90 minutes.

[tp1de]

Thanks for testing. For me it might look like that the influxdb handling might have changed within ioBroker. I recognized some other changes in function with newer js-controller. I am changing some code in the moment - not related to energy calculation.

[Curufinwem]

So the old records from earlier today stayed correct but the new one from this evening got duplicated again:

[Curufinwem]

But this one is not in the db yet

The JSON does not hold a record for 20:00 but for 21:00: [ { "date": "2023-10-09 21 hrs", "val": 0 }, { "date": "2023-10-09 19 hrs", "val": 0 }, { "date": "2023-10-09 18 hrs", "val": 3.78 }, { "date": "2023-10-09 17 hrs", "val": 0 },

Let's see what happens at 20:00

[tp1de]

I identified that the ioBroker db functions do not work as expected with influxdb v2 anymore. I was not able to make them work until now. I will continue tomorrow.

[Curufinwem]

Interesting, the database now has a duplicate:

And the JSON did not change at this time:

[Curufinwem]

I identified that the ioBroker db functions do not work as expected with influxdb v2 anymore. I was not able to make them work until now. I will continue tomorrow.

Great, thanks. Let me know if I can help or should try out some thing on my end.

[tp1de]

I can't delete any InfluxDB records for one measurement within my adapter coding. How do you delete the records for one measurement?

[Curufinwem]

I can't delete any InfluxDB records for one measurement within my adapter coding. How do you delete the records for one measurement?

Yes, this is a bit unintuitive. I did it via console in the end. Iirc, you first have to add a config with a token from influx: influx config create -n iobroker -u http://localhost:8086 -t my_token_here -o my-org

Then you can delete a single measurement with with this: influx delete --bucket ioBroker --start '1970-01-01T00:00:00Z' --stop $(date +"%Y-%m-%dT%H:%M:%SZ") --predicate '_measurement="ems-esp.0.energy.actualPower._Hours"'

[tp1de]

Let me share my findings in respect to InfluxDB V2:

InfluxDB V2 has one global retention policy. For reading the historic consumption data from km200 the rp has to be set > 2 years. Individual policies per state are not possible. Within ioBroker deletion of data is not supportet. The internal getHistory functions give different results than for mySQL/MariaDB. The aggregation functions per intervall are not reliable. This makes a global adapter supporting multiple databases extremly difficult.

I recognized that data inserts are extremly slow compared to mySQL which is more than 100 times faster. I tried to fix the coding and released V 2.5.0 as a temporary solution. (I hope this will avoid duplicates)

The needed polling cycles will insert new values in the database every 15 seconds. This will result in 2 mio records p.a. No problem for mySQL and I can delete not needed records within the adapter. For InfluxDB this is not possible.

I need to review if I still should support InfluxDB.

[Curufinwem]

Thanks a lot for your efforts! I will try out the new version after work today.

It seems this causes a lot of trouble and I have to say I am not hellbent on the usage of influxdb. When I set up my home automation stuff this was just one of the first things that popped up and for most usages it is fine. I am not really familiar with the syntax and I also find stuff like the deletion of records cumbersome. This is probably just missing experience but since my knowledge of SQL is somewhat better the easiest option might be to just switch to a MariaDB. My setup is also using Proxmox so adding a container with the different database is not difficult and I do not think reading from that DB into Grafana will cause issues.

So if influx costs too much of your time and is not really used, I can totally see dropping the support for influx making sense.

[tp1de]

Influx has recognized that version V2 was not easy to use and the own query language was too difficult for most users. Version 3 will be rolled out soon and a new ioBroker adapter supporting Influxdb V3 as well. Influx promises 45 times faster write access !

I tested Influxdb V1 and V2 with home assistant but finally went back to use MariaDB. Anyhow I prefer ha compared to iobroker especially in respect to the energy dashboard. Since some weeks the ems-esp energy consumption can be seamless integrated there as well.

The ioBroker adapter is now used by > 1200 users. I am willing to support for some period. But for the ems-esp energy data I need to change to the new energy entities from ems-esp rather than calculating myself.

[Curufinwem]

That is probably a good choice. Let's see if v3 is a bit more friendly at entry level.

Maybe I will give HA a spin. I like IoBroker for the automation stuff but I do not see why they should not be run in parallel in case the different tasks are split up nicely.

Off-topic - just to make sure I understand: ems-esp now provide energy data on its own?

[tp1de]

Off-topic - just to make sure I understand: ems-esp now provide energy data on its own?

Yes, after implementing the energy consumption statistics within ioBroker, I proposed to implement the logic within ems-esp firmware - version >= 3.6.2 needed ! I'ts basically an energy counter - one for ww and one for heating - calculating the energy consumption in kWh. There are 3 entities within boiler:

nompower - If not set automatically in your boiler you can set manually nrgheat and nrgww the consumption fields for warm water and heating.

In iobroker you should find them under boiler or if you selected km200 like structure they are under heatSources.hs1. .....

[Curufinwem]

Ahh, also very interesting. I will check this out. Thanks for the info!

[Curufinwem]

So I updated the version late last evening and had to wait for today but it looks good now:

With regards to the number of records there seems to be some type of aggregation. I can only see 11 records since midnight for the _hours measurement.

Great work, thanks a lot!

[tp1de]

With regards to the number of records there seems to be some type of aggregation. I can only see 11 records since midnight for the _hours measurement.

As I tried to explain: The calculation is done in 2 steps:

1. Write raw data every 15 seconds into the power state (multiplier of modulation * nominal power)
2. Aggregate the data to _Hours, Days, Months - these are database fields for use of Flot or other graphs

(1) will be 2 mio records p.a.

Btw: Have you found the nrgheat / nrgww states?

[Curufinwem]

Ah yes, my mistake. Sorry, I did not read thoroughly enough. The power measurement does indeed have a lot of records.

Yes, I found them. I also started record the history of them in influx.

Very small divergence in numbers but it is negligible and I assume due to rounding, difference in calculation methods and / or time intervals.

[tp1de]

If I compare ioBroker to home assistant I can recognize that the hourly values have a time difference of 2 hours. ioBroker is 2 hours earlier. I believe that this is the difference between UTC and CET. I have to check the coding.

Could you please check your timestamps if it's the same for you?

[Curufinwem]

Not sure if I understood correctly but I think the timestamps are fine for me.

These are my hourly values:

And these for comparison my burner percentages:

They are aligned and it is times I would expect the heating to start up again in the morning, or, in this case, when the temperature dropped below the defined threshold for warm water in the morning heating period. I can double check with the burner around 18:00 when the warm water for the evening should start.

[Curufinwem]

I vaguely remember having some issues with the times in ioBroker when I started out a few months back. I think I had to set a timezone on the container. I seem to have some time adjustment shown in the system information:

[tp1de]

Thanks for checking ... the mistake is in my km200 part for hourly recording values and not in the ems-esp energy statistics. Ems-esp is fine. I will release a new adapter version for km200 soon.