Feature request: Opt-in for long-term statistics

[chvancooten]

Hi, thanks for the great component! It's helping me a lot with trying to get the best out of my dynamic energy contract.

To perform some data analysis on historic pricing, energy cost, etc., it may be beneficial to opt-in the "current price" exposed by this integration for long-term statistics as described here. This should be a pretty small change, and would allow Hass to collect aggregate data for longer than the 10-day default period.

Thanks again for your work on this!

[JaccoR]

Hi, thanks for the great component! It's helping me a lot with trying to get the best out of my dynamic energy contract. Thanks again for your work on this!

No problem! Thanks for testing it out.

To perform some data analysis on historic pricing, energy cost, etc., it may be beneficial to opt-in the "current price" exposed by this integration for long-term statistics

Is the energy cost not saved in the energy dashboard longer than 10 days now? The attributes are pretty big, and when recorded for longer periods of time take up a lot of space in the DB. For home assistant instances running on a small rpi this could be a problem. It is indeed a small fix, so if people would support this I can add it.

[chvancooten]

Currently, the values are managed only by the 'recorder' component which stores all the data for 10 days by default. You can check that the price sensors are not managed as a long-term statistic by adding a new 'Statistics Graph' card and looking for the sensors in the drop down, they won't show up there.

With regards to the size (and write frequency in the DB), Hass solves this for statistics-enabled sensor by only storing one aggregate record per hour for min-mean-max values for the sensor state itself (24 entries per day). For the current price sensor that doesn't matter much tbh, since it's only one value per hour anyway. I think other attributes (e.g. the full details for today/tomorrow) are discarded automatically. It doesn't really make sense to store statistics data for the "helper" sensors either, since that data can be derived from the price itself.

[JaccoR]

Thanks for the insights, did not know home assistant handled it that way. I will look into this!

[Mariusthvdb]

The attributes are pretty big, and when recorded for longer periods of time take up a lot of space in the DB.

this in itself is a bit of an issue really. Reason for core devs to deny this usage in core integrations, (eg check the discussion we had on the Here travel time route) and breakout attributes to single sensor entities.

For the Entso-e integration: why populate all sensor entities with the full attribute? There is no need for that? Why not create those single entities, and keep the full data array inside the integrations internal code.

At least take out the 'prices_today' array from all sensors. but, also the other attributes: It's odd to have all sensors states as attributes in all other sensors.

btw,I hacked them out with:

        invalid_keys = {"time_min", "time_max","prices","prices_today","prices_tomorrow",
                        "avg_price","max_price","min_price","current_price",
                        "next_hour_price"}

but that also takes away the full data set for the graphs.... guess we need a dedicated sensor entity for that, or maybe some other way of creating these entities in the first place. Because now, keeping those attributes in the self._attr_extra_state_attributes

creates them on all sensor entities.

[JaccoR]

You're right. I don't know of a way to make the attributes entity specific. Is that possible in home assistant? self._attr_extra_state_attributes indeed changes all entities attributes.

I am still questioning if creating sensors for every hour is a good idea, there seem to be some issues with using attributes... Making graphs would be more complicated though.

[Mariusthvdb]

just a thought (for the time being ...): create an extra sensor, or separate the 'prices' sensor, and add the complete attribute to that entity only.

That way all other entities could simply be that, and dont carry all of the identical attributes (which, as said, are simply the states of the other attributes.) we're going in circles now, and the DB doesnt like that, and is filed with cross references and huge attributes.

ofc, its a custom_component, and you are in no way obliged to follow HA guidelines. That's the prerogative of a custom component. However, its used in larger HA setups, and these guidelines are set for a reason. So if at all possible, I believe it would be awesome if you could do that.

Maybe ask Frenck what he thinks? He's on top of these things in HA dev team.

[JaccoR]

just a thought (for the time being ...): create an extra sensor, or separate the 'prices' sensor, and add the complete attribute to that entity only.

So you mean add an entity which only carries the attributes? Then it would still be necessary to give different sensors different attributes right? I am not sure if that is possible.

ofc, its a custom_component, and you are in no way obliged to follow HA guidelines. That's the prerogative of a custom component. However, its used in larger HA setups, and these guidelines are set for a reason. So if at all possible, I believe it would be awesome if you could do that.

I would definitely follow HA guidelines! I completely agree and I would follow them wherever possible. However, it is unclear how HA guidelines would handle this situation. Would you say it is HA guidelines to create separate entities for all 48 hours?

[Mariusthvdb]

well, I fear so.... check buienradar, it has 108+ entities per station.... but dont take my word for it please, I am no dev! I guess you'd best consult with Frenck (he pointed me to your integration ;-) ), or check in one of the Discord #devs channels, maybe #devs_energy or just #devs

So you mean add an entity which only carries the attributes? Then it would still be necessary to give different sensors different attributes right? I am not sure if that is possible.

well, if you'd have 1 mother of entities.... all the other entities wouldn't even need attributes any longer, because like currently, they are all simply the values of the entities right?

current_price, next_hour etc etc, would remain standalone entities, but these values are not required as an attribute on any of the other entities?

[JaccoR]

I see what you mean. I will contact Frenck and get back to this!

[JaccoR]

I am currently working on getting one sensor to carry all attributes. I would say this should be current price. Any thoughts on this?

[sfstar]

It sounds logical to have current entity carry the attributes. However, it and next hour (etc) are hourly updated. Perhaps it would then be better to couple it to a sensor with less frequent updates (like the high or low entities) so that the DB is even less taxed. (If you were to go this route don't forget to update the apexchart in the readme to target the new correct entity carrying the attributes).

[JaccoR]

If we were to go with a sensor with less frequent updates i would say average price. Another option would be creating a new entity especially for carrying the attributes. I am not sure if this would be clearer or cause more clutter.

[sfstar]

Indeed average price is also an good option as a less frequently updated entity. Not creating another entity would ensure that all entity primary values have a purpose. On the other hand having a specific entity with the attributes will also allow users to specify to not have that entity recorded (via exclusion). However, these attribute values would only (with the current implementation) change every 24 hours so there is not a lot DB clutter if everything is saved 365 times a year.

So my personal preference would be to have it combined with an existing entity (until HA future values support is here). Because I believe an entity primarily should have a state and optional attributes. Which would not be the case for a dedicated entity unless you were to set an attribute value as it's state.

[JaccoR]

My preference also goes to adding it to an existing sensor. For this average price would be the best option imo. I'll add this in the new commit.