Feature request: Time-shifting energy consumption

[magnusoverli]

It would be great if this integration could aid users when trying to time-shift energy consumption towards the cheapest hours of the day. Maybe even let uers tailor a template they can use in automations.

This can be a complex matter, because the end-users needs are very different, based on household, geography, personal preference and such, but I have an idea on how to let users easily get going..

Say I wanna try to shift the times when my water boiler consumes energy. I estimate that my water boiler needs to have access to energy at least 13 hours of the day, to make sure we don't get any nasty bugs in the water. So, if this integration can provide a binary sensor that allows the user to give it a name and define the amount of hours needed (X), it could evaluate to true only during the X amount of hours that are the cheapest. It would be easy to use directly in time-shifting consumption for the water boiler.

I also have an EV. On any regular day I would not need more than 5 or 6 hours of access to energy to charge the car. I could define a new binary sensor that is only true during the 5 (or 6) cheapest hours.

Of course, people are starting to really invest in solar panels, and that propably further complicates things, but lets tackle one problem at the time. Right now, I cannot see any easy way to get started with (more or less) intelligently time-shifting energy consumption... This would be a huge deal!

[jpulakka]

I have done something related https://github.com/jpulakka/nordpool_diff . Currently it uses https://github.com/custom-components/nordpool to fecth prices but I'm planning to support entso-e at some point https://github.com/jpulakka/nordpool_diff/issues/21

It's not exactly what you want, though. It was originally intended to adjust air heat pump target temperature, and the output can be thresholded to make it somehow work with e.g. water boilers, but there's no direct way to say "give me 5 cheapest of the following 15 hours".

[JariInc]

I've been thinking this also and here's my two cents:

1. Rank, just rank the hours from cheapest to most expensive. Then you could do something simple as rank <= 13 to get the 13 cheapest hours. Of course this would not take account tomorrows prices which could be even cheaper.

2. Normalized price, basically highest price would be 1 and lowest would be zero and everything else between those. This would allow you to pick, for example, the cheapest quartile. It wouldn't guarantee you certain amount of hours but the cheapest hours. Let's say you have EV and you don't need it to charge fully everyday, you could just charge it with cheapest 20% hours. Some day this could be just 1 hour, some other day 12 hours.

Looks like @jpulakka has these on his nordpool_diff repo already.

There's no upper limit how complex you can make this but I would start with the simplest things.

[JaccoR]

This is a pretty neat idea. I am doubting a bit if this would surpass the purpose of this integration. I kind of feel like this integration would just provide the prices, so users can tinker with it as they wish. They can use for example the integration of @jpulakka. I don't want to unnecessarily complicate things. What do you think, should it be part of this integration?

[JariInc]

This is a pretty neat idea. I am doubting a bit if this would surpass the purpose of this integration. I kind of feel like this integration would just provide the prices, so users can tinker with it as they wish. They can use for example the integration of @jpulakka. I don't want to unnecessarily complicate things. What do you think, should it be part of this integration?

As a user I would appreciate if the integration would give me some basic statistical properties. As a developer I appreciate the Unix philosophy, do one thing and do it well.

This didn't answer your question, I know :grin:

[jpulakka]

Rank, just rank the hours from cheapest to most expensive. Then you could do something simple as rank <= 13 to get the 13 cheapest hours. Of course this would not take account tomorrows prices which could be even cheaper.

Yep, people often want to know "X cheapest hours" but it's not that easy to define what exactly that means.

• X cheapest hours "today" probably already passed during the night.
• X cheapest upcoming hours "today" is well-defined, but ends arbitrarily at 24:00.
• X cheapest upcoming hours during the next Y hours is also well-defined, but the solution changes by time and particularly in the afternoon when tomorrow's prices become available.
• So, X cheapest upcoming hours during the next Y hours (or today) now is not necessarily the same time range as X cheapest upcoming hours after 1 hour from now. In an appropriate edge case (continuously falling prices) the cheapest hours are fleeing into the future hour by hour and you never get to charge your EV.
• Also, should the hours be contiguous (required to run for example washing machine) or is it ok to have breaks between them?

What comes to nordpool_diff, it uses a sliding window looking (only) at the next filter_length hours. It has pros and cons, but at least it's well-defined. As you say, there's no upper limit how complex you can make this :)

---

Regarding Unix philosophy vs. bundling "some basic statistical properties" in the same integration, that's a question of taste, but extra features in one integration is a rabbit hole and they often end up being a bit half-assed.

The challenge with Unix philosophy in Home Assistant is that there isn't (as far as I know) a standard way to move complex information, such as "price as function of time" between integrations. So each upper layer needs to be customized to be able to use information from the lower layer (e.g. https://github.com/jpulakka/nordpool_diff/issues/21) in whatever particular format the lower layer provides.

[MrOzzOnline]

This is a pretty neat idea. I am doubting a bit if this would surpass the purpose of this integration. I kind of feel like this integration would just provide the prices, so users can tinker with it as they wish. They can use for example the integration of @jpulakka. I don't want to unnecessarily complicate things. What do you think, should it be part of this integration?

From a users point of view it's important there is enough added value from the integration to make it interesting. Piecing some kind of solution together as a HA newbie or a user who isn't familiar with the development tools is such a steep learning curve. People will just give up.

A useful integration at least needs:

• A list of the cheapest hours within a specific time frame which can span over a day. Maybe you could add a config setting from what time onward the next day needs to be included in the evaluation. (for example, I charge my car between 7pm and 7 am)
• The time frame should be manageable by a (weekly?) schedule similar to a thermostat
• The entities (or any other solution to make the data available) should be easy to use in automation, templates, node red, ...
• Bonus: comparing prices of a longer period (eg: weekday, weekend, week, month)

An example, my situation (some stuff still planned for the next few weeks): dynamic hourly electricity pricing, EV charing 30-40kWh each night, 22kW solar panels, 44kWh battery. I would like to be able to manage things for example: charge untill the home battery is xx%, only charge when I inject power, only charge during negative prices, only charge during the cheapest x hours at night ect ...

FYI One more thing to add: the Belgian government just decided to give a €8.000 tax cut for people buying a bidirectional charger. There is going to be a high demand for automation based on the hourly prices.

[jpulakka]

A list of the cheapest hours within a specific time frame which can span over a day.

But the devil is in the details. That list defined at 12:00 today is different from that list defined at 18:00 today. And as explained above, in an appropriate edge case (continuously falling prices) the cheapest hours are always in the end of the time frame and "now" is "cheap" only just before the prices start rising again, which could take arbitrarily long time (even though typically that happens in the morning).

Maybe you could add a config setting from what time onward the next day needs to be included in the evaluation. (for example, I charge my car between 7pm and 7 am)

This would solve the problem. "N cheapest hours between 7pm and 7am" is well-defined (but some applications might require a contiguous range of hours while others are ok with possible breaks between the hours).

[MrOzzOnline]

A list of the cheapest hours within a specific time frame which can span over a day.

But the devil is in the details. That list defined at 12:00 today is different from that list defined at 18:00 today. And as explained above, in an appropriate edge case (continuously falling prices) the cheapest hours are always in the end of the time frame and "now" is "cheap" only just before the prices start rising again, which could take arbitrarily long time (even though typically that happens in the morning).

I think most (if not all?) private contracts work with day ahead prices. The prices for the next day are typically known around noon or even a bit more early. Either way, I would suggest handling predictions for continuous changing prices would out of the scope of this integration. It just needs to provide the prices at the moment the information is downloaded using the api, it doesn't have to make predictions. That is a whole other thing.

Maybe you could add a config setting from what time onward the next day needs to be included in the evaluation. (for example, I charge my car between 7pm and 7 am)

This would solve the problem. "N cheapest hours between 7pm and 7am" is well-defined (but some applications might require a contiguous range of hours while others are ok with possible breaks between the hours).

Defining several ranges during a day could fix this: for example the cheapest price for the next 3 hours, the cheapest price between xx and yy, and aa and bb, limited by the availability of the data.

[jpulakka]

handling predictions for continuous changing prices would out of the scope of this integration

I didn't mean that we should "predict"/guess anything, apart from what https://github.com/JaccoR/hass-entso-e/issues/3 already does (returns prices for as far to the future as ENTSO-E API provides),

My point is that even if you know all the prices a day or even more ahead, deciding which of those are "cheapest hours" is not a trivial problem and it can be answered in many ways depending on the exact definition. Some boundaries need to be defined as you propose, e.g.

• N cheapest hours between 7pm and 7am. (Answer to this changes in the afternoon when more data becomes available, but that's ok since the change is known well before 7pm.)
• Cheapest single hour within the next N hours
• Cheapest contiguous average between xx and yy

But already there we have three different things! How to express all this in compact way is a challenge. I have followed HA forums etc. quite a bit but haven't seen a clear solution so far.

[MrOzzOnline]

• N cheapest hours between 7pm and 7am. (Answer to this changes in the afternoon when more data becomes available, but that's ok since the change is known well before 7pm.)
• Cheapest single hour within the next N hours
• Cheapest contiguous average between xx and yy

But already there we have three different things! How to express all this in compact way is a challenge. I have followed HA forums etc. quite a bit but haven't seen a clear solution so far.

Not a bad solution?

I've been thinking this also and here's my two cents:

1. Rank, just rank the hours from cheapest to most expensive. Then you could do something simple as rank <= 13 to get the 13 cheapest hours. Of course this would not take account tomorrows prices which could be even cheaper.
2. Normalized price, basically highest price would be 1 and lowest would be zero and everything else between those. This would allow you to pick, for example, the cheapest quartile. It wouldn't guarantee you certain amount of hours but the cheapest hours. Let's say you have EV and you don't need it to charge fully everyday, you could just charge it with cheapest 20% hours. Some day this could be just 1 hour, some other day 12 hours.

Looks like @jpulakka has these on his nordpool_diff repo already.

There's no upper limit how complex you can make this but I would start with the simplest things.

So, if you work this out you would have the following sensors:

• array current day values (not sure if it's possible to add an array to a sensor?)
• array next day values
• current price
• current rank (1 to max) - max is the highest value within the considered range
• current normalized price (0 to 1)
• current contiguous average price
• current contiguous average rank

Maybe you could have the same options for the next hour too

Some configuration options I can think of:

• relative or absolute hourly range
• relative range (in case of relative addressing)
• start & end hour (for absolute range)
• Contiguous range
• update frequency?
• API stuff

[magnusoverli]

Lots of good input here! And a few concerns that are absolutely valid!

I just want to be clear about my motivation for this FR. I already have access to prices for my region, including the next day (as soon as those are decided). I have been playing with templates to sort hours based on price (not unlike some of the ideas here). The trouble is getting those templates right. It is a fair bit of work, but absolutely possible! Not for the average user though… And that is important to remember.

So, for many users, I think access to prices are not the main concern, but how to use that data in automations. Being able to bridge the gap between the price data and usable automations is where the “gold mine” lies, so to speak… ☺️

[blowk]

FYI One more thing to add: the Belgian government just decided to give a €8.000 tax cut for people buying a bidirectional charger. There is going to be a high demand for automation based on the hourly prices.

@MrOzzOnline: where can I find this information?

In Belgium there are companies that provides smart devices that controls the battery and taking in account the solar forecast, dynamic prices and energy prediction based historical user data from the digital meter.

In HA we already have an energy dashboard, solar forecast and energy data. I don't know if HA has any plans to apply dynamic prices on that so automations can rely on this data?

[MrOzzOnline]

FYI One more thing to add: the Belgian government just decided to give a €8.000 tax cut for people buying a bidirectional charger. There is going to be a high demand for automation based on the hourly prices.

@MrOzzOnline: where can I find this information?

Article in Dutch: https://www.hln.be/geld/forse-belastingvermindering-op-komst-voor-laadpaal-die-in-twee-richtingen-stroom-levert~aafe0cbd/

TL;DR; you'll get a tax reduction up to €8.000 for a bidirectional charger starting from 1/1/2023

[jpulakka]

@MrOzzOnline

array current day values (not sure if it's possible to add an array to a sensor?) array next day values current price

Current price is the main output of this integration. Current & next day values are already available (as attributes) in https://github.com/JaccoR/hass-entso-e/issues/3

current rank (1 to max) - max is the highest value within the considered range current normalized price (0 to 1) current contiguous average price current contiguous average rank

These require configuration options, as you have listed. It would be important to define those in more detail.

@magnusoverli

access to prices are not the main concern, but how to use that data in automations. Being able to bridge the gap between the price data and usable automations is where the “gold mine” lies, so to speak

Exactly!

From this perspective, there are two particularly useful outputs.

• Continuous number: how cheap or expensive it is now, in some arbitrary but predictable scale, so that thermostat setpoint can be adjusted around the default.
• Binary boolean: is it "cheap" or not now. One way to define this is to threshold the continuous output.

https://github.com/jpulakka/nordpool_diff does this. But it doesn't do constraints or requirements needed for some real-world use cases, such as

• Binary output for EV charging must be on at least X hours between 7pm...7am
• Binary output for water heater must be on at least 2 hours during every 20 hours (even if prices are falling continuously, so even if "now" would be the most expensive hour for 20 hours in a row; unlikely, but possible)

I think defining how exactly to express constraints like the above is the missing part. If we can figure that out, the remaining computation part is easy. Whether that should be part of this integration or kept separate is up to @JaccoR ...

[MrOzzOnline]

One of the biggest or difficulties is the availability of the data. At some point during the day, the batch of data for the next day will become available. This could change the evaluation of cheap / expensive power completely.

So, in response to @jpulakka I suggest the following configuration options I've posted before. I'll explain them in more detail:

• relative or absolute hourly range: Here you can choose if you want to evaluate the price rank for a specific fixed (= absolute) range or for the next xx hours.
• relative range (in case of relative addressing): just the next xx hours
• start & end hour (for absolute range): a fixed range for example: between 7pm and 7am. Bonus: weekly schedule so you can have different options during the weekend
• Contiguous range: This is the average price evaluated over a specific time frame. For example: a range of 2 will calculate and rank the prices over a block of 2 hours, a range of 5 will do the same in blocks of 5 hours. This option can be used if you need to pull power for a continuous period which is over 1 hour long.
• update frequency?: not sure if this is needed. The data only changes once each day. Maybe this is needed when pulling the data fails for some reason (no internet connection for example). How much time should the integration wait before it tries again?
• API stuff: api token
• (NEW) Time when the next day is available: hh:mm time when the next day data is available from the api.

One more thing which should be decided on is the behavior of the integration when it ranks prices when data is only prially available. For example: How do you rank data for a 7pm - 7am timeframe during the time of the day when only the 7pm to 12pm data is avaliable? There are 4 options I can think of:

• trigger some data unavailable error for the sensor
• rank only the data for the hours available (in this example, rank 7pm -12pm from 1 to 5)
• rank the data assuming the unavailable hours are more expensive as the unavailable hours.
• rank the data assuming the unavailable hours are cheaper.

What do you guys think. I think the first option might be the most easy to implement, but are there any common problems where the other options are needed?

[jpulakka]

@MrOzzOnline Good thinking, thanks for describing what the options mean!

A couple of thoughts:

• Someone might want one set of rules to control water heater, another set for controlling EV charging, etc. So it might be useful to enable N sets of these options, instead of just global options.
• What exactly is the output signal, and where & how would you convert that into a control signal that says "on" or "off"? You have mentioned sensors such as "current rank", should that then just be thresholded or what?

One of the biggest or difficulties is the availability of the data. At some point during the day, the batch of data for the next day will become available. This could change the evaluation of cheap / expensive power completely. ...One more thing which should be decided on is the behavior of the integration when it ranks prices when data is only prially available. For example: How do you rank data for a 7pm - 7am timeframe during the time of the day when only the 7pm to 12pm data is avaliable?

In practice this less of a problem than in theory, because:

• At least 10-12 next hours are available any time. It's rarely necessary to know at 10am the exact sheduling details of 7pm-7am next night. We just need to accept that the schedule (if we would even define it before 7pm) might get fine-tuned in the fly. This becomes more of problem only if a schedule changes while we're already executing it. Even then:
• The nearest unknown hours are right after midnight, which is almost always "cheap" compared to the evening hours. So, before enough data is available, heuristics such as "pad using the last known price" (nordpool_diff does this) or "pad using yesterday's data from the same time" or "rank the data assuming the unavailable hours are cheaper", as you proposed, are reasonable. This is a low-hanging and almost correct way to avoid having to handle missing data in the calculations.

[MrOzzOnline]

A couple of thoughts:

• Someone might want one set of rules to control water heater, another set for controlling EV charging, etc. So it might be useful to enable N sets of these options, instead of just global options.

It would be nice to have multiple sets, but it would make it (a lot?) more complex

• What exactly is the output signal, and where & how would you convert that into a control signal that says "on" or "off"? You have mentioned sensors such as "current rank", should that then just be thresholded or what?

Use it in automations, node red or other integrations (like https://www.home-assistant.io/integrations/threshold/)

The rank lists the current price from 1 (cheapest) to maximum of the range within a specific range. In my example is you want to charge your car during the 5 cheapest hours between 7pm and 7am you set this range an monitor the sensor. If it's value is below 6, start charging.

[jpulakka]

It would be nice to have multiple sets, but it would make it (a lot?) more complex

Not really, that requires just not conflating things into big ball of mud. Two things are needed:

• There's a way to enter multiple parameter sets. I don't know if/how this could be done with config flow, but with configuration.yaml (https://github.com/JaccoR/hass-entso-e/issues/18) you simply create several sensor entities, each with different settings, and that's it. I think it's more common than not to do that.
  • BTW this is one argument for keeping "get electricity price from somewhere" vs. "create control signal from the electricity price" integrations separate. Probably nobody needs to know prices from >1 countries (so it should be sufficient to enter API key and Area only once), but as said, it would be very nice to process the price in >1 application-dependent ways.
• The entity ID has a settings-dependent part, so that it remains unique and it's possible to see which of them is which. For example https://github.com/custom-components/nordpool creates entity ids like "sensor.nordpool_kwh_fi_eur_3_095_024" where price_type (kwh), region (fi), currency (eur), precision (3), low_price_cutoff (095) and VAT (024) are part of the ID.

In my example is you want to charge your car during the 5 cheapest hours between 7pm and 7am you set this range an monitor the sensor. If it's value is below 6, start charging.

Ok, I understand the idea. For this to work the rank should output "maximum" outside the 7pm...7am range, or consider all hours outside the range "infinitely expensive".

[JaccoR]

These are all very nice ideas! We should definitely work on something like this. However, this will turn this integration into some kind of Energy Management System. I think that is not really the purpose of this integration. It should just provides energy prices from ENTSO-e in my opinion.

I am currently working on an EMS for my job, so i would definitely want to be working on an EMS system with pricing inputs for home assistant. I think we should start another project for an integration that takes price (and things like PV) as inputs and you can configure the output sensor the way you like as mentioned above. I can make a different repository for that where we can continue this discussion if you agree.

[magnusoverli]

I think that is a great idea! I am no developer, but will for sure keep a close eye on a new project! FWIW; I'd really like to test along the way also...

[jpulakka]

I agree it's better to keep things separate, so ENTSO-E provides energy prices, to be consumed by other integrations.

That said, there already is something built-in native https://www.home-assistant.io/home-energy-management/ , then there's a really ambitious-looking optimizer https://github.com/davidusb-geek/emhass , and lots of smaller random integrations. It would probably be a good idea to first study what the already available features/integrations do. If none of them do what's needed, then we need one more integration :)

[JaccoR]

That said, there already is something built-in native https://www.home-assistant.io/home-energy-management/ , then there's a really ambitions-looking optimizer https://github.com/davidusb-geek/emhass , and lots of smaller random integrations. It would probably be a good idea to first study what the already available features/integrations do. If none of them do what's needed, then we need one more integration :)

The first one you mention is just the energy dashboard and its attributes. It is not like an energy management system at all imo. You can not control you battery / loads with it. It could however be inputs into a real ems system.

Ive tried using EMHASS, but i thought this was a way too complex and felt very messy and cumbersome. It felt like you needed to take part in the development of the integration to be able to use it. I eventually gave up using it, so I am not sure what exactly it is capable of though. I don't know about the smaller random integrations.

I am really just thinking about an integration where you can give a couple of inputs (pricing, energy demand, start/end time, type of load etc), the algorithm crunches the numbers in the background and it will give you the sensor you need to control your load (EV, heatpump etc) using automations. At least, thats how I see it. I can not find anything that does something similar.

[jpulakka]

Ok, HASS built-in "EMS" seems to be focusing more on dashboards and monitoring rather than controlling things.

Good to hear some experience about EMHASS. It seems impressive at first glance but I never tried it. Indeed it seems complex, and having HASS to run shell scripts & get results with curl from localhost is... an interesting approach.

Philosophically I'm in the same boat with simple integration that you propose. https://github.com/jpulakka/nordpool_diff gives contiguous output signal which is IMO good for tuning thermostats, and although it can be thresholded to control binary loads, it's not ideal for that. There's no way to define "N cheapest hours between X and Y" with it, and I have no plans to add that.

So, go for it!

[MrOzzOnline]

I'm currently using this integration. It's good to calculate your costs and to draw nice graphs, but the real benefit is an EMS like you are talking about. I understand you want to keep this separate, but the current available sensors don't provide the needed information which you need in to build an EMS in a separate integration or some other solution (nod red, templates, ...) You need at least:

• an array with price information for the current day
• an array with price information for the next day
• bonus: a ranking (1 to 24 for both)
• super bonus: a ranking spanning 2 days I'm not sure what would be the best way to do it? A sensor with attributes? It would make a massive difference if you could provide this within this integration.

I agree an EMS could be very complex. I'm wondering which things an EMS integration could do better? You already have automation's which could link everything together.

[jpulakka]

@MrOzzOnline As mentioned above, hass-entso-e already provides

• an array with price information for the current day
• an array with price information for the next day

That was done in https://github.com/JaccoR/hass-entso-e/issues/3

With that information, everything that a separate EMS integration needs, is already available from hass-entso-e.

[MrOzzOnline]

Yeah, I just saw (still learning sorry).

It would be very nice to have ENTSO-E integrated in https://github.com/jpulakka/nordpool_diff

It doesn't rank the prices in a fixed time frame as @jpulakka mentioned before, but it goes a long way. You can use automations an multiple sensors to get around this

[jpulakka]

It would be very nice to have ENTSO-E integrated in https://github.com/jpulakka/nordpool_diff

@MrOzzOnline Yep, there's https://github.com/jpulakka/nordpool_diff/issues/21 about that. Should be quite straightforward but I'm too busy with everything right now... Will get back to that.

Also looking forward for @JaccoR 's

I am really just thinking about an integration where you can give a couple of inputs (pricing, energy demand, start/end time, type of load etc), the algorithm crunches the numbers in the background and it will give you the sensor you need to control your load (EV, heatpump etc) using automations. At least, thats how I see it. I can not find anything that does something similar.