Suggestion on pre_start and reference periods.

[mdallaire]

The pre_start period should probably be a separate setting in the config.ini to let people set it to what makes sense for them.

I also think adding the reference periods as described by Blobule in the HQ forums (unfortunately there is no direct link to comments available but a search for "11h-14h" should get you to the right one) would be a great addition. I am not sure if it's best to add it as static values or config setting with a default.

I did not know about reference periods last year (I just reduced my consumption during peaks) but I started using it in my automations this year and the difference in credits is major for me (last winter 76$ total credits, this year 35$ with just the first 4 periods!) for the same reduction in usage in peak periods.

[zepiaf]

That's valid anything static is bad anyway. I'll het a look at the comment you mentioned as i did not see it

[zepiaf]

I just reread, and i realized that the ref period is exactly the pre start. As per hydro doc on the system the reference period starts 3h before the event hence my setting at -3h. Still it can be configurable and defaulted to -3h

[mdallaire]

Can you link the document? I am trying to find it on hydro's website but can't find any that talk about the reference period.

[clauderobi]

But the post is talking about 1-4h and 11-14h. Not 3-6h and 13-16h. So there is a disconnect. Where is the Hydro doc? May be put a link here.

Also, do you make a distinction between pre-start on a day without a peek event and pre-start on a day with a peek event?

[zepiaf]

la somme des moyennes des valeurs réelles enregistrées à chaque heure durant les périodes 33

de 3 heures consécutives débutant 5 heures avant toutes les plages horaires pertinentes de la période de référence, soit celles de la nuit ou de l’après‐midi, selon que l’événement de pointe critique a lieu le matin ou le soir, exclusion faite des valeurs minimale et maximale de chaque heure.

Section 8

https://www.hydroquebec.com/data/documents-donnees/pdf/tarifs-electricite.pdf#page=36

I suppose there is an english version somewhere but i hope you guys can manage in french ;)

[mdallaire]

Sans problème, avec Olivier, Mathieu et Claude comme prénoms, il devrait pas y avoir de problème avec des documents français 😉

Reading this I think there are two approaches to the reference period.

Either go with something like "start_ref" and "end_ref" or "ref_offset" and "ref_duration".

The first one is probably the easiest to automate with but the second one gives some flexibility if hydro changes their calculations down the road.

[mdallaire]

I have been thinking about this a bit more and I think combining the two would probably be best. Something like this:

Both of these could come from the config since they can't be accessed from the API, (as far as I know). "ref_offset":5 "ref_duration":3

Then the following times can be calculated: "start_ref"= "next_occurence_start_time"-"ref_offset" "end_ref"= "start_ref"+"ref_duration"

[zepiaf]

That is doable however i'd rather use two offsets :

ref_start_offset:-3 ref_end_offset:0 (That would be equivalent to my setting)

That way you avoid duration issues making the ref period overlap with the event.

To feed the discussion here is my logic in homeassistant:

-3h(pre_start): raise temp event_start: lower to predefined values event_end: restore default temp

These are my 3 automations + 3 to feed datetime inputs (helpers) from mqtt

Mqtt is for the time being fed by my publisher ran as a cronjob at 10 past each hour

[mdallaire]

Edit: I changed "reference" periods to "anchor" period to align with the terms we now use in the integration.

Ok I think we don't see the anchor period the same way.

I see you use it as a pre-heat period before lowering the setpoint for the peak period to keep the house warmer during that time.

My understanding (from Blobule's comment I referred to earlier and the documentation from Hydro you provided) is that the anchor period is a time where Hydro's algorithms compare the energy you usually consume during that time frame with the energy you consume just before the upcoming peak and use it as a part of their calculations to determine the "Énergie effacée".

In my case here is what happened today before tonight's peak:

11h

• Start charging the EV
• Do laundry that needs hot water or the dryer
• Run the dish washer
• Set the heat a bit higher
• Cook some food in the oven
• Etc

14h

• Stop charging the EV if I don't need it later today (I will start charging again at 1AM if there is a peak tomorrow morning and the car is not full)
• Regular energy usage
• Leave the heat higher since there is a peak coming

The idea is to use more energy than usual in that time frame so that Hydro's algorithms believe that the cold weather is really having us heat a lot more than usual. In my case I have some stuff automated and a notification to me and my wife that now is a good time to do some of those power hungry things. It's not really about using more energy than usual but moving stuff that would happen at other time during the day/night to that specific time frame to boost the winter credit.

Here is what I have for a recent period:

Dates et heures	Énergie de référence	Énergie réelle	Énergie effacée	Crédit
4 janv. 2022, de 16 h à 20 h	33,12 kWh	3,44 kWh	29,67 kWh	15,02 $

Énergie de référence :Estimation de l’énergie habituellement consommée dans la même plage horaire qu’un événement de pointe. (This is the definition from the "Legende" of the website)

I can assure you that I rarely (probably never) consume 33 kWh between 16h and 20h. Average is ~10kWh with some exceptions to 15-20 kWh if I charge the car because I need to go out in the evening which is probably once or twice a month. So the extra energy used during the anchor period helps raising the "Énergie de référence" in the above table.

The anchor period is really a time frame in Hydro's calculations of the credit. You can still set a pre-heat period that overlaps the anchor period to raise the heat at 13h and finishing at 16h. You would then have 1h (13h-14h) of additional consumption in your anchor period that help with the credit calculation.

You can really push the whole thing and try to reduce your consumption during anchor periods on non-critical peak days to further increase the "Énergie de référence" more. The optimal formula would be to move energy consumption during normal days from the anchor periods to the peak periods as much as possible, basically doing the reverse of a critical peak day.

I hope it makes thing clearer?

[zepiaf]

Yeah i think i oversimplified, rereading the doc from hydro plus your explanation, it's starting to make sense. It's way more complicated than i thought. I'll try to model it in the output and keep it configurable.

[clauderobi]

I think what you do with the reference period and how you calculate them are 2 different topics. The thing that calculates the reference periods must not need to understand the purpose of the various periods. On some day, you must turn On the heat during the reference period. On some other days you must turn Off (as per the post Mathieu referred to). So, keeping the 2 logic (defining the periods vs what to do with them) separate is best; and use neutral naming too.

[clauderobi]

If, according to the doc, the reference periods start 5h before the peak event and last 3h, what does the "pre_start" field in the JSON data means? The reference are 1h-4h et 11h-14h while the pre_start fields are 3h and 13h; is pre_start just the time when the app should send a reminder to the user?

[mdallaire]

zepiaf set the prestart as a way to pre-heat before the peak period. That is why I made a lengthy post last night to try to better explain the concept of the reference period to him. I think he will try to change the logic when he gets more time to work on this.

[clauderobi]

That is was I understood too. I am just talking about the JSON output when we run the tool. I am assuming it is just a dump of the what HQ is sending but I guess I was be wrong....

In an earlier post in this thread I suggest to keep the period definition logic and period usage logic separate because as you indicated in your post last evening sometime it is used to pre-heat or to reduce electricity consumption depending on factors that are outside the strict definition of the period boundaries.

BTW, may I suggest to not use human readable representation of time but rather use time since epoch (in seconds). Having to convert back and forth every time the value is used is unnecessary when you only need the string representation when displaying on a screen of sort. Also, a string time representation without the time zone information is not great.

[zepiaf]

Pre_start should be pre_heat, allows for better comfort and reduce consumption by inertia.

This should be configurable.

What is needed is :

1. current_status: [reference,pre_heat,normal,peak]
2. Config option to format datetime (epoch/string format): I use this format because homeassistant is working with it, @clauderobi it seems you're using something different.
3. peak_tomorrow: true/false
4. Allow configuration of pre_heat

As for the doc my understanding is slightly different, it is not only on the day of the event but rather an average of the 5 similar days before (last 5 weekends days or last 5 weekdays) + the period of 3h starting 5h before the event.

According to this and assuming the events are always starting at 6am or 4pm, any 1am to 3am and 11am to 2pm should be a ref period, solving all problems (as we don't know much in advance if there will be a peak or not)

[mdallaire]

I read your initial post and your other thread talking about separating the components and it all make sense. But since this project is 4 days old and 1 developer we will need to be patient. The way things are presented right now is that the API extraction is built and there are two sample scripts provided (mqtt.py and hydro.py) for us to play around, build on top of and eventually submit PR.

There are definitely more architecture decisions to be made.

Regarding the human readable representation of time, I tend to agree with you that having unix time sent is a good thing, but the current format is meant to be sent to Home Assistant input_datetime set_datetime service so I would argue it is not useless.

[mdallaire]

To expand on this:

Pre_start should be pre_heat, allows for better comfort and reduce consumption by inertia.

This should be configurable.

What is needed is :

1. current_status: [reference,pre_heat,normal,peak]

2. Config option to format datetime (epoch/string format): I use this format because homeassistant is working with it, @clauderobi it seems you're using something different.

3. peak_tomorrow: true/false

4. Allow configuration of pre_heat

As for the doc my understanding is slightly different, it is not only on the day of the event but rather an average of the 5 similar days before (last 5 weekends days or last 5 weekdays) + the period of 3h starting 5h before the event.

According to this and assuming the events are always starting at 6am or 4pm, any 1am to 3am and 11am to 2pm should be a ref period, solving all problems (as we don't know much in advance if there will be a peak or not)

I think we could structure the data like this:

current_period: possible values:

• reference (every day 11h-14h and 1h-4h)
• peak (every day 6h-9h and 16h-20h)
• normal (every thing else)

current_period_status: possible_values:

• critical/winter peak
• normal

The combination of both of these in automations would then give us the "moment" value. The current_period start/end could be config variables for which hydro's values are provided but can be modified if for some reason someone wish to do so.

upcoming_criticalpeak:

• true/false

This could be set as soon as we know a critical peak is coming, this would address scenarios like @clauderobi's where he want to start some automations ASAP when there is a peak coming.

[zepiaf]

I'm working on a few things:

• Separating winter credit calculations in another module (high level / low level)
  • Low level being hydro_api raw stuff
  • High level being wintercredit calcs (next event, ref_period, current state, pre heat)
• Reformatting the object to only output meaningful data
• Updated config to yaml as ini files are not the future :)

[clauderobi]

Cool. As I read it, current_period_status looks like it is saying if today is a peak day or a normal day. If so it is missing the fact that a peak day may have a peak in the morning only, afternoon only (like today) or both. If this is the case, then the name should be changed for current_day_type and would be valid for a given day (not the day before) (as indicated by the name saying current and day).

OTOH, current_period_status may be simply an instantaneous state saying peak / non peak, which is not enough. In such case, I would add 2 more states:

• Reference normal - For non peak days
• Reference peak - For peak days

I would also rename current_period for current_period_type.

Thanks for upcoming_criticalpeak, which could be active the day prior but does not participate in the current_xxxx of this day.

[mdallaire]

Cool. As I read it, current_period_status looks like it is saying if today is a peak day or a normal day. If so it is missing the fact that a peak day may have a peak in the morning only, afternoon only (like today) or both. If this is the case, then the name should be changed for current_day_type and would be valid for a given day (not the day before) (as indicated by the name saying current and day).

The way I see it "current_period_status" would be based on the next "ultimate event" which is the state (critical vs normal) of the next peak, it would be set right after the last peak period and would reset after the next peak period.

So lets take today's example : Critical peek period between 16h and 20h	time	current_period	current_period_status	comment
midnight - 12h59	normal	normal	nothing special happening
1h-3h59	reference	normal
4h-5h59	normal	normal
6h-8h59	peak	normal	peak period that is non-critical
9h-10h59	normal	critical	the next reference and peak periods will be critical
11h-13h59	reference	critical	reference period preceding the upcoming critical peak
14h - 15h59	normal	critical
16h - 19h59	peak	critical
20h - 1h	normal	normal	when the last peak end the status is reset to normal unless the next peak is critical

My approach would be that the current_period_status bit would be set right after the last peak period and apply to all upcoming periods until the end of the next peak period. The code already have a way to determine the existence of a "next" critical event.

I like having the two information separate as I can check for a simple true/false of current_period_status instead of checking for all possible combination of period and status but your approach could also work and could look something like this:

time	moment	comment
midnight - 12h59	normal-normal	nothing special happening
1h-3h59	reference-normal
4h-5h59	normal-normal
6h-8h59	peak-normal	peak period that is non-critical
9h-10h59	normal-critical	the next reference and peak periods will be critical
11h-13h59	reference-critical	reference period preceding the upcoming critical peak
14h - 15h59	normal-critical
16h - 19h59	peak-critical
20h - 1h	normal-normal	when the last peak end the status is reset to normal unless the next peak is critical

One way of doing it does not exclude the other and it is probably easy enough to provide the two approaches.

Thanks for upcoming_criticalpeak, which could be active the day prior but does not participate in the current_xxxx of this day.

I was actually thinking that upcoming_criticalpeak would be triggered to true as soon as Hydro publishes the information and remain true until there is no more future peaks announced.

[clauderobi]

I agree about separation. And it should be clear that current_period is just a "dumb" status without logic. While in the case of current_period_status it depends if you want to separate the "dumb" peak / non peak vs more complex logic, such as the logic in your table. I prefer the former as I suspect you do.

In the first table, the period from 1h - 3h59 has no comment while in fact you want no heating at that time since there is no peak at 6h-9h. If it were that a peak would be active at 6h-9h then you want to pre-heat during that period as you want today at 11h-13h59.

Yes, I understood upcoming_criticalpeak the same way you describe.

[clauderobi]

I reread again, I guess current_period_status is correct. It probably just need to be clear that essentially it is about the next upcoming peak period being critical or normal as you described in you very last comment. Just nomenclature. as usual.... ;-)

[zepiaf]

EDITED

That's what i have now :

For the current_state it is either : normal / reference_period_morning / reference_period_evening / peak

NEXT_EVENT = 
{
 "date": "2022-01-10",
 "start": "2022-01-10 16:00:00",
 "end": "2022-01-10 20:00:00",
 "start_ts": 1641848400.0,
 "end_ts": 1641862800.0,
 "pre_heat_start": "2022-01-10 13:00:00",
 "pre_heat_end": "2022-01-10 16:00:00",
 "pre_heat_start_ts": 1641837600.0,
 "pre_heat_end_ts": 1641848400.0
}

STATE = 
{
 "state": {
  "current_state": "reference_period_evening",
  "event_in_progress": false
  "pre_heat": true,
  "event_today": true,
  "event_tomorrow": true
 },
 "reference_period": {
  "morning": {
   "date": "2022-01-10",
   "start": "2022-01-10 01:00:00",
   "end": "2022-01-10 04:00:00",
   "start_ts": 1641794400.0,
   "end_ts": 1641805200.0
  },
  "evening": {
   "date": "2022-01-10",
   "start": "2022-01-10 11:00:00",
   "end": "2022-01-10 14:00:00",
   "start_ts": 1641830400.0,
   "end_ts": 1641841200.0
  }
 }
}

[clauderobi]

I am not sure event_today is just a 2 state affair; I think 4 states are needed to differentiate between days with 2 events or just an event in the morning or in the afternoon.

For example, during the reference_period_morning time, one would have the combine that current_state info with NEXT_EVENT to make a decision, which is certainly doable. But we would need to know what is the type of NEXT_EVENT, unless they exists only for an active peak event and not as per Mathieu comment where the NEXT_EVENT simply tells us if the next peak period is active or inactive.

But then event_today purpose is not clear.

[zepiaf]

so basically :

today_morning_event: true / false today_evening_event: true / false tomorrow_morning_event tomorrow_evening_event: true / false

?

[clauderobi]

Correct.

[zepiaf]

https://github.com/zepiaf/hydroqc/pull/17

It's quite a change, sorry I hate massive PR.

@clauderobi, @mdallaire if you want to get a look, otherwise I'll just merge and we'll bugfix on the go :)

[mdallaire]

@zepiaf You forgot to add config.yaml to your .gitignore

[zepiaf]

DAMAGE_CONTROL_MODE : True

[zepiaf]

Where were we ?

https://github.com/zepiaf/hydroqc/pull/18

And thanks for spotting :D

[zepiaf]

I merged the PR @mdallaire @clauderobi satisfied it resolves the issue ?

[clauderobi]

Look good.

Purely cosmetic, I would have used negative number for the offsets. This is easier for the reader.

[clauderobi]

Now that we are not during a peak period, I see that you drive the value for morning_event_today (and I guess the other similar keys) based not only that there is or not a event today but also what time of day are we (i.e are we currently after the event). Normally, for this type of keys, which are reporting a status I would prefer to do so only for 1 source of information and not combine multiple sources into a composite status.

You can still have composite status, such as pre-heat, as needed but the raw status should be available to. The composition logic may be different for different use cases.

[zepiaf]

@clauderobi can you expend a bit more on the composite status you're talking about, i'm not sure i'm following you there. I'm happy to make the change once I understand properly :) Maybe an example ?

[clauderobi]

Ok.

morning_event_today was true early today. It is now false. But yet, today, there is a morning event even though it is past. My composite I mean that morning_event_today the merge of 2 status: the fact that Hydro declared an event this morning AND the fact this it is up coming or past. I am not saying that the composition is good or bad, I am just saying that my doing a composition you are assuming that the caller has the same needs. By providing the raw status you enable the caller to implement its own logic. This, obviously, does not preclude to provide a composite status as a separate key.

[zepiaf]

I'll try to keep the past events. The reason why it's going to false is that the event is marked as past once it happend. Let me check if I can change the logic so that events are in the past once the day is over. Would that work ?

[clauderobi]

well, then it is not morning_event_today but yesterday or another key (which I do not see in the output).

[zepiaf]

what I'm saying is not change morning_event_today until the day's over (basically change once today is yesterday)

that way it would stay to true till midnight then change according to the new day.

I think that's what you said right ?

[zepiaf]

PR #20 is changing that so morning and evening events will stay true even once they have completed and until the day is over

[mdallaire]

I think there is a flaw in the logic for the current-state. Looking at the code (with my very minimal python knowledge) at lines 80-81 of winter_credit.py it looks like current_state:peak will only happen when there is a critical peak event. I can't really confirm it since we only had critical peaks since the code was published so I may be wrong.

Since hydro also refers to previous non-critical peaks in their calculations it would make sense to have the peaks in the current_state even when there is no critical peak events. The first table in this comment may help clarify it.

This means the current_state (current_period in the table) could be static (or in config.yaml) in the winter_credit.py logic and don't really depend on the hydro API.

The event_in_progress works well with peak events but don't help in indicating if a "reference" current_state is part of (just before) a critical peak event. I think we could still benefit to have a secondary "event" related status that cover the status or the periods between two peeks (current_period_status in my other table).

The logic would be something like this: Right after the last peak period we check the status (critical or non-critical) of the next peak and apply the value right now to the "current_period_status", this way we know whether the next periods (normal, reference) are "pre-critical peak" or not and we can apply the various automation logics based on this.

To be honest I already am doing all this in node-red/home-assistant right now, I just use the winterpeaks/next/start|end values to get the critical peak status. But I think having it published in mqtt would be valuable and will allow people to work with different automation logics.

[clauderobi]

@zepiaf, this is correct. Tomorrow is a new day and morning_event_today takes whatever value corresponding to what Hydro publishes. But, what happens if Hydro is late (due to some uncontrolled situation) and declare a peak event after midnight? I know that this probably never occurs but what if?

[clauderobi]

@mdallaire I was thinking like you and would want to have the presence of peak event periods regardless if they are active or not and have such active or not data added to the JSON output. But, these info, morning_event_today and evening_event_today gives you the same information.

[zepiaf]

@zepiaf, this is correct. Tomorrow is a new day and morning_event_today takes whatever value corresponding to what Hydro publishes. But, what happens if Hydro is late (due to some uncontrolled situation) and declare a peak event after midnight? I know that this probably never occurs but what if?

Hydro has a policy saying peak events are advertised before 5pm the day before. I honestly think that trying to put contingency for that is overkill :)

[zepiaf]

@clauderobi @mdallaire : regarding the current_state I think indeed it should be peak during peak periods and that the event_in_progress should give if it's actually an "event"

[mdallaire]

On a side note, I think we should define a naming scheme or at least a dictionary of terms and what they mean in regards to Hydro winter peaks so that we are all on the same page and make thing easier to figure out for newcomers.

I suggest we use the verbiage already used by Hydro in their documentation. I will try to find an English version to the one @zepiaf linked higher up, put it in a markdown file and submit a PR to discuss it.

[zepiaf]

https://www.hydroquebec.com/data/documents-donnees/pdf/electricity-rates.pdf?v=20210517#page=35

[zepiaf]

we must add the peak periods to the config.yaml as well not just the start time, that way i'll be able to update the current state to peak - criticalpeak - ref[morning|evening] - normal

[mdallaire]

Ok initial winter credit readme is created: #21 @zepiaf and @clauderobi feel free to comment over there!

[mdallaire]

If you are interested in the exact formula behind the winter credits it is very well described here: http://publicsde.regie-energie.qc.ca/projets/469/DocPrj/R-4057-2018-B-0062-DDR-RepDDR-2018_10_26.pdf#page=127

[mdallaire]

Do we consider this issue resolved?