Potential issue with Intelligent Dispatching Sensor & Price reverting to peak rates

[Leventionz]

Describe the bug

Please see this topic for details:

custom_components.octopus_energy.coordinators.electricity_rates

Reproduction steps

Must be on Intelligent Octopus Plug in car Wait for Intelligent Dispatching Sensor to turn "ON" If car is charged during this period Completed Dispatches attributes is updated. If the car does not get charged for a variety of reasons (OE doesn't initiate charge, external setting prevents charge etc) the time period in question does not show up in Completed Dispatches but Intelligent Dispatching Sensor remains "ON"

Expected behaviour

The main priority is to understand if when Intelligent Dispatching Sensor is "ON" irrespective of Completed Dispatches does the price remain at off-peak or does it revert to peak pricing. Without my bill which I will get in a few weeks it is difficult to verify this.

If it does remain at off-peak pricing when Intelligent Dispatching is enabled then the current accumulative cost sensor should be updated to reflect off-peak period throughout and not revert back to peak pricing even if there was no Completed Dispatches.

It is entirely possible that OE are aware the car is not charging and as such whilst Intelligent Dispatching is enabled the Completed Dispatches correctly identifies only those sessions when the car was charging and reverts to peak pricing even though the car is plugged in. In such a case there is no bug.

Tariff Code

E-1R-INTELLI-VAR-22-10-14-M

Integration Version

8.2.0

Home Assistant Version

Home Assistant 2023.9.2

Fresh Install?

Not specified

Home Assistant Logs

home-assistant_octopus_energy_2023-09-20T12-45-05.155Z.log

[BottlecapDave]

I'll await your future confirmation. Either way, we should be able to update the integration to reflect the behaviour of OE (e.g. store the pending dispatches once it has been entered for future calculations if this is indeed the behaviour)

[Leventionz]

I got my bill through finally; I evaluated the period of the 19/09/23 when my car was plugged in but not charging between 9.30am and 12pm.

Completed dispatches on that day shows:-

start: '2023-09-19T14:30:00+00:00' end: '2023-09-19T15:00:00+00:00' charge_in_kwh: -0.26 source: null"

The bill shows I have been billed the entire period at off-peak on this date.

09:00 - 09:30 7.14 0.01 0.071 09:30 - 10:00 7.14 1.37 9.814 10:00 - 10:30 7.14 1.91 13.657 10:30 - 11:00 7.14 1.94 13.886 11:00 - 11:30 7.14 1.92 13.707 11:30 - 12:00 7.14 1.22 8.743

Based on this example I think as long as the car is plugged in even if it not charging you still get off-peak pricing.

Therefore the "bug" is reverting the Current Accumulative Cost sensor to "peak rates" when "Completed Dispatches" is not present. Planned Dispatches alone should be enough to give confidence of "off peak" pricing.

I would like to hear other IO users views on this as well.

I can also confirm separately that every time 'Completed Dispatches' is populated the price is also off-peak (I assume OE provide this data because they can see actual car specific charging)

[BottlecapDave]

Thanks for the update in information. The completed dispatches is only available via the API for a short amount of time (I believe testing had them disappear within 6 hours). To counteract that, I save the completed dispatches locally for up to 3 days to accomodate the default previous consumption sensor. However, there's a chance that the API could be unavailable for whatever reason and the integration fails to download and save the completed dispatches in time, which could have occurred in this scenario.

Have you seen this occur at any other point?

[Leventionz]

Hi, just to be clear my observations indicate 'Completed Dispatches' is not necessary to benefit from the off-peak rate. As long as the car is plugged in and 'Intelligent Dispatching' is 'on' then you will get the price at off-peak rate in my experience.

I understand you are using 'Completed Dispatches' as a way of calculating the 'Current Accumulative Cost' true cost but I do not think this is necessary.

[BottlecapDave]

I'm using completed dispatches as a way of knowing if the car was plugged in (as there doesn't seem to be anything else in the API) and therefore entitled to the off peak rate. To my knowledge, if a pending dispatch is provided outside of the normal off peak period and the car isn't plugged in, then the rest of the house isn't entitled to the off peak rate? So the completed dispatches is the only way of knowing this.

[Leventionz]

I'm using completed dispatches as a way of knowing if the car was plugged in (as there doesn't seem to be anything else in the API) and therefore entitled to the off peak rate. To my knowledge, if a pending dispatch is provided outside of the normal off peak period and the car isn't plugged in, then the rest of the house isn't entitled to the off peak rate? So the completed dispatches is the only way of knowing this.

Thanks for the explanation Dave. In my research it doesn't matter if completed dispatches occurs; the only thing that matters is that the car is plugged in.

Therefore another way to calculate this would be to just assume when the Intelligent Dispatching sensor is ON you are in receipt of the off-peak rate and planned dispatches is enough to obtain the look ahead period. If you unplug your car during a planned dispatch period then only the periods that your car was connected count, the Intelligent Dispatching sensor gives this information anyway in the sensor history (off-peak = when sensor is on, peak = sensor off). As this sensor only activates when the car is connected it should be sufficient to provide you with a method of calculation.

Perhaps you could build into the integration an option to chose the calculation method between completed dispatches and the method I suggest above; then the user can pick based on their confidence level of my analysis.

Separately, I would like to thank you for your time and efforts with this integration, it is highly appreciated and is saving me a lot of £.

[andynash]

I am seeing the same thing (car plugged in but not charging, Intelligent Dispatching on, period(s) do not show in Completed Dospatches), but I'm not yet in receipt of a relevant bill, so I'm not able to confirm that these periods are indeed proved at off-peak rates. I am hoping so as it will otherwise make a significant difference to the bill.

I see the cumulative cost sensor leaping up at certain points, appearing to suddenly decide to charge these periods at peak rate (after the fact) because they have not appeared in Completed Dispatches.

I am currently relying on the Inteligent Dispatching sensor to trigger other loads, e.g. dishwasher, battery charging, so incurring (according to the cumulative cost sensor) significant peak rate costs. I believe and hope though that this is actually not the case and these periods remain off-peak.

The recalculation appears to happen when the car is unplugged prior to the charge-by time and therefore during a smart-charge period (and possibly on other occasions too) - I guess at that point Octopus 'finalise' the session and the Completed Dispatches triggering the integrations recalculation of cumulative cost?

Am I correct in thinking the integration is making this cumulative cost calculation? Previously I had assumed the cumulative cost figures came direct from the API without manipulation, but reading above it seems not?

I think such situations have arisen before and have been billed off-peak, therefore I expect/hope the next bill will confirm this 🤞

That said, I wonder if in the future Octopus might use Completed Dispatches for billing if they can get it working the way they want? That would allow them to apply off-peak strictly only when the car is charging (not just plugged in).

Even if you adjust the integration to calculate cumulative cost as is currently (hopefully) being calculated, it may be worth bearing this in mind in case the current behaviour is needed again in the future?

[BottlecapDave]

Lot to unpack here, so please bare with me

Therefore another way to calculate this would be to just assume when the Intelligent Dispatching sensor is ON you are in receipt of the off-peak rate and planned dispatches is enough to obtain the look ahead period.

This integration is in charge of when this sensor goes on/off and this is how it currently works

1. If you're during an off peak period. This comes straight from the tariff data and is what also powers the off_peak sensor
2. You are in a planned dispatch (which could be within the off peak period or outside). There could be a potential issue here, where I'm not filtering out planned dispatches where the location isn't home, but that shouldn't be effecting you.

Another thing that I couldn't get to the bottom of during the beta was when a pending dispatch disappeared. It is possible that a pending dispatch is being removed from the API results before it has finished (or started) which would cause the sensor to turn off. I'm also not sure if once a pending dispatch has been set, if it could be unset (e.g. they set it for 10-11am but then need to change it to 11-12 due to grid needs)

As this sensor only activates when the car is connected it should be sufficient to provide you with a method of calculation.

The API provides no information as to if a car is attached and the integration doesn't know this either. That's why I said it was best guess that a planned dispatch is going to provide you the cheaper rate. There is a status field available in the API, but as I'm not on intelligent I'm unable to investigate the value of this during various stages. I could look at exposing this in a future release to determine this information.

[BottlecapDave]

I see the cumulative cost sensor leaping up at certain points, appearing to suddenly decide to charge these periods at peak rate (after the fact) because they have not appeared in Completed Dispatches.

Rate information (which is used to calculate consumption cost) uses both the planned (for present and future) and completed (for past) dispatches to adjust any rates that are peak rates to the off peak rate. This is indicated in the raw data via the is_intelligent_adjusted field. However like you said, there's a chance that a pending dispatched doesn't turn into a completed dispatch and so that rate would go back up, which would cause the cost to be evaluated at the higher price.

The recalculation appears to happen when the car is unplugged prior to the charge-by time and therefore during a smart-charge period (and possibly on other occasions too) - I guess at that point Octopus 'finalise' the session and the Completed Dispatches triggering the integrations recalculation of cumulative cost?

I believe that if the car is unable to charge during a pending dispatch and it is outside of the off peak period, then the off peak rate will not be applied. But again, I'm not on this tariff so not sure. The unknown status field might be able to give better indications of this.

Am I correct in thinking the integration is making this cumulative cost calculation? Previously I had assumed the cumulative cost figures came direct from the API without manipulation, but reading above it seems not

You are correct that the integration performs all of the calculations, including the intelligent adjustments that occur outside of off-peak times.

[BottlecapDave]

I've posted in the OE forums for some clarification

[github-actions[bot]]

This issue has become stale because it has been open for 30 days with no activity. If you still think it's an issue, please respond soon.

[Leventionz]

I just wanted to provide input to you based on my experience now having a second monthly bill come through.

Whenever the intelligent dispatching sensor is "on" I am only being charged 7.5p

Therefore I have created my own accumulative sensor which more accurately aligns to the correct price when using IO tariff which simply assumes when intelligent dispatching sensor is "on" the rate is off peak else it is the peak rate. Now my bill is very closely aligned to what is reported in HA.

[BottlecapDave]

I've had a few reports that the completed dispatches are only used in certain scenarios (I think ohme). Therefore I plan on recording and saving the planned dispatches locally to more accurately curate a bill. If a planned dispatch disappears before it starts then I won't save it. If it disappears while it's started then I will. Can anyone confirm if planned dispatches disappear after they've started but before they've ended to confirm this logic?

I'm also going to add a disclaimer that this sensor is "best effort" based on the data available and is at the users own risk when on outside off peak period as the sensor doesn't know about other factors that might stop them getting the cheaper rate (e.g. car not charging).

I wish there was something a little better in the APIs that could be relied on.

[Leventionz]

I am not understanding something here clearly but however you are identifying the intelligent dispatching sensor state 'on' or 'off' when it is 'on' could you not just set the price to the off peak rate, and when it is 'off' to the peak rate?

or am I oversimplifying? this is what I am doing with my custom sensor and it's working well.

[BottlecapDave]

The rates need to be adjusted for both the future (target rate sensors) and past (consumption cost sensors)!as well as now. There is a central process that retrieves and post processes rates (because OE doesn't do this for you as part of the API) and the sensors just read from this central process (HA recommended architecture)

[Leventionz]

Understood, I only need the current and then use that for my accumulative sensor calculation which is why I can use the on/off state to help determine the current pricing.

[andynash]

I can also confirm that for current pricing, the Intelligent Dispatching sensor accurately reflects whether a slot is at peak or off-peak rate.

I also think planned dispatches do disappear as described, but will keep a closer eye on them...

Thanks for your work on this Dave!

[MikeShipp]

Octopus produced a schedule last night ending at 6am. When the car was charged, it actually ended at 5:30am (the car was still able to accept charge, so not stopped by me but by OE revising schedule, I guess). The "Octopus Rates" card confirms 5:30 to 6:00 was a peak period but "dispatching" only went off at 6:00. I use "dispatching" to control my batteries to prevent them being discharged to EV during bonus slots. I actually charge the batteries whenever "dispatching" is on but, as the batteries are full, it doesn't have a major impact.

[andynash]

Octopus produced a schedule last night ending at 6am. When the car was charged, it actually ended at 5:30am (the car was still able to accept charge, so not stopped by me but by OE revising schedule, I guess). The "Octopus Rates" card confirms 5:30 to 6:00 was a peak period but "dispatching" only went off at 6:00. I use "dispatching" to control my batteries to prevent them being discharged to EV during bonus slots. I actually charge the batteries whenever "dispatching" is on but, as the batteries are full, it doesn't have a major impact.

I guess until you get your bill you won't be able to confirm 100% whether 05.30 to 06:00am was peak or off-peak?

My working theory is that if any part of any half hour is (smart-charge) off-peak, the whole of that half hour will be billed at off-peak rates. I also think this works both forwards and in arrears, I.e. plug in at 23:12 and 23.00 to 23:30 will be off-peak (for everything). If your car finishes charging at 05.33 and 05:30 to 06:00 was a planned smart-charge dispatch, then everything will benefit until 06:00, even though the car battery is full. But I'm not 100% sure of that yet.

In your case, if the planned dispatches really were amended part way through the night and ended at 05.30 then the dispatching sensor is, in theory, incorrect (which is odd as it is surely based on planned dispatches and the rates card should be based on the same info?), but I'm curious what originally suggested the car charging ended at 5:30? (You said the car could have taken more charge, the rates card confirmed it was peak from 05:30 to 06:00, and the dispatch sensor disagreed, I think?)

[MikeShipp]

Octopus produced a schedule last night ending at 6am. When the car was charged, it actually ended at 5:30am (the car was still able to accept charge, so not stopped by me but by OE revising schedule, I guess). The "Octopus Rates" card confirms 5:30 to 6:00 was a peak period but "dispatching" only went off at 6:00. I use "dispatching" to control my batteries to prevent them being discharged to EV during bonus slots. I actually charge the batteries whenever "dispatching" is on but, as the batteries are full, it doesn't have a major impact.

I guess until you get your bill you won't be able to confirm 100% whether 05.30 to 06:00am was peak or off-peak?

My working theory is that if any part of any half hour is (smart-charge) off-peak, the whole of that half hour will be billed at off-peak rates. I also think this works both forwards and in arrears, I.e. plug in at 23:12 and 23.00 to 23:30 will be off-peak (for everything). If your car finishes charging at 05.33 and 05:30 to 06:00 was a planned smart-charge dispatch, then everything will benefit until 06:00, even though the car battery is full. But I'm not 100% sure of that yet.

In your case, if the planned dispatches really were amended part way through the night and ended at 05.30 then the dispatching sensor is, in theory, incorrect (which is odd as it is surely based on planned dispatches and the rates card should be based on the same info?), but I'm curious what originally suggested the car charging ended at 5:30? (You said the car could have taken more charge, the rates card confirmed it was peak from 05:30 to 06:00, and the dispatch sensor disagreed, I think?)

So far, I've only looked at the rate card for early slots. I've usually forgotten the next day as the usual issue was that Zappi failed to charge the car. That appears to be over. Not long to wait for the bill (maybe).

[BottlecapDave]

Hello, sorry for the late response.

The dispatching sensor works from the pending dispatches provided by OE. These are updated frequently, but there is a chance that the schedule could be updated on OE end and the integration doesn't reflect that (e.g. unable to get the updated schedule). This would be reflected in the logs.

The rate information, which the rates card is probably fed from, is built by stitching various bits of data - The tariff rate information, pending dispatches (which adjust present and future rates to off peak rates) and completed dispatches (which adjust past rates to off peak). Therefore if you look at the rate card retrospectively, the rates might be different to what was present at the time because the completed dispatches can be a subset of the pending dispatches (as they might not have been valid). However, its best to note that I've had a sprinkling of reports that completed dispatches have had off peak billed slots missing.

Without concrete account, billing and API data it's hard for me to go to OE with issues in what's reported by the API.

[MikeShipp]

Hi Dave

I eventually got the last bill (I had to ask but got it within minutes of asking!) and the low tariff rate appears to follow the dispatching signal for the entire month and includes the period for which the rate card was in conflict (I was charged at the lower rate, as indicated by "dispatching").

Just for clarification, the car didn't charge in the period in question, 5:30am-6:00am, but stopped at 5:30pm. Dispatching and the bill show the lower tariff between 5:30 and 6:00 and the bill also shows no significant energy consumed.

Thanks for your work on this integration.

[github-actions[bot]]

This issue has become stale because it has been open for 30 days with no activity. If you still think it's an issue, please respond soon.

[github-actions[bot]]

This issue has been closed because it has been inactive for 14 days since being marked as stale. This is done to help keep on top of active issues. If you still think it's an issue, please respond to this issue

[andynash]

Not sure if this should be a separate issue, but I'm not 100% clear where we ended up with this.

In particular, when a half hour session is off-peak, but then part way through (for example if the car leaves home earlier than expected), Pending Dispatches are removed I think? When I last checked this, the Dispatch sensor was then turning off immediately, not at the end of the half hour slot.

I think (based on bills, but also logically given Octopus can only bill at a single rate per half hour slot) that session remains off-peak until the half hour boundary is reached.

Looking at yesterday, there appears to be a completed dispatch for the partial half hour during which the car left, which appears to corroborate this (I think times are now in local time?).

The car left around 17.10ish, but the Dispatch sensor remained on until 17.28:00 so it looks as though the Dispatch sensor now possibly matches my expectations, unless the 17.28:00 end time means something else is at play (e.g. the car was late reporting its departure to Octopus)?

[BottlecapDave]

Please update to v10.1.3. If there is no delay between the pending dispatch being removed and the completed dispatch being added, then the dispatch sensor should now continue until the end of the half hour (as it uses the same data as the rate sensor)