Previous Accumulation Sensors not reseting at Midnight

[gb095666]

Describe the bug

Home Assistant 2023.8.4 Supervisor 2023.08.1 Operating System 10.5 Frontend 20230802.1 - latest Octopus Installed version v8.2.0

The Current accumulation Sensors seem to reset at midnight thanks to the latest update, However the data is not moved over to the previous accumulation sensor entities at midnight.

Reproduction steps

The system should do this automatically at midnight

Expected behaviour

At midnight the data from the "Current Accumulation Sensors" should be copied into the equivalent "Previous Day Accumulation Sensors", and then the "Current Accumulation Sensors" should be reset to zero ready for the current days usage/cost.

Tariff Code

No response

Integration Version

8.2.0

Home Assistant Version

2023.8.4

Fresh Install?

Not specified

Home Assistant Logs

Not relevant

[BottlecapDave]

Hi there. This is currently by design as the sensor is used by those both with and without a Home Mini. It is currently feeding data from the main APIs, which is shown in the app when looking at historic data. This data can differ from the data reported by the Home Mini, and is used by OE to create your bills, and is used as the source of truth over the mini.

For instance, I show both the current and previous consumption sensors on my energy dashboard so I can get the live readings on the day, but the "real" readings when looking back at the data. I feel that if I change this, I will get similar reports from users that it doesn't match the data in the app. Therefore, at the current time I will not be changing this behaviour.

[smithbill17]

I've been looking at my Energy figures and started to realise, exactly as @gb095666 said when he opened this issue, that the Previous Accumulative sensors don't update just after midnight as you would expect. Mine seem to update at 7am or 8am in the morning & for gas sometimes update more than once in a day.

But the reason for posting, is I don't understand the explanation for why it is like this? Isn't it logical that at midnight, today's consumption becomes yesterday's consumption??

@BottlecapDave said "if I change this, I will get similar reports from users that it doesn't match the data in the app." Why wouldn't it match the data in the Octopus app? Surely if I consume 10kWh today, then at midnight the 10kWh moves over to yesterday in both the app & the Octopus Mini data?

[BottlecapDave]

But the reason for posting, is I don't understand the explanation for why it is like this? Isn't it logical that at midnight, today's consumption becomes yesterday's consumption??

Why wouldn't it match the data in the Octopus app? Surely if I consume 10kWh today, then at midnight the 10kWh moves over to yesterday in both the app & the Octopus Mini data?

As mentioned,

This is currently by design as the sensor is used by those both with and without a Home Mini. It is currently feeding data from the main APIs, which is shown in the app when looking at historic data. This data can differ from the data reported by the Home Mini, and is used by OE to create your bills, and is used as the source of truth over the mini.

I believe the app behaviour has changed since I last made this post. However, if I look at my OE dashboard on the website, I can only see data up to friday

This is backed up further, by me hitting the API that returns my electricity data for yesterday, which shows bringing back zero data

If I look at fridays data within HA energy dashboard, you can see that there's discrepencies between what the Home Mini reported I used and what my meter said I used

In this case, it's only a tens of watts, but if I look at the day before, the discrepency is a lot higher

It is these discrepencies, coupled with HA sensors not built to change historic data which is why the sensors don't "switch over" at midnight. As anyone who looks at their dashboard retrospecitvely, if I showed solely home mini data, would raise bugs stating the calculation was incorrect.

[smithbill17]

I really appreciate all the effort you put into this integration - it's brilliant. So I really don't mean to be 'obtuse', but I'm still not 'getting it'.

It's obviously something to do with there being some significant discrepancies between HomeMini supplied data & the data the Octopus app uses - but why is there a discrepancy? Surely it's my SmartMeter originating the data for everything, so why is there a discrepancy?

I'm guessing when you say "if I showed solely home mini data, would raise bugs stating the calculation was incorrect" then you're saying the data from the SmartMeter is what's shown in the Octopus app & the data from the HomeMini, isn't?

But back to my original query - when I look at the data supplied by my HomeMini to your integration sensors & compare it to my Octopus app (after accounting for StandingCharge & VAT), it matches. Great! So why then, when we get to midnight, doesn't it rollover the correct consumption for today, into yesterday? Instead, the data is updated by HomeMini around 7am with a figure that doesn't quite match what the SmartMeter reported throughout the day yesterday?

I'm obviously missing something. I guess it means I just can't rely on the data in the 'Previous Accumulative Consumption' sensors being accurate.

[BottlecapDave]

It's obviously something to do with there being some significant discrepancies between HomeMini supplied data & the data the Octopus app uses - but why is there a discrepancy? Surely it's my SmartMeter originating the data for everything, so why is there a discrepancy?

Why is there a discrepency between the data the the mini reports to OE and what your meter reports to OE? For that, you would have to consult OE as I just get the data from their APIs.

I'm guessing when you say "if I showed solely home mini data, would raise bugs stating the calculation was incorrect" then you're saying the data from the SmartMeter is what's shown in the Octopus app & the data from the HomeMini, isn't?

This is not what I'm saying. From what I can understand, your smart meter doesn't send your data in real time, nor does it send it straight to your supplier but instead via the DCC. The retrieval of data from the DCC by OE is not done in real time and is also delayed. This is why if you don't have a home mini it can take up to 24 hours (or sometimes more) to see the previous days worth of data.

The home mini is a bridge for this delay, and kind of acts as an alternative to the real time display that a lot of people have in their homes. However, I believe OE have to generate your bills by what the DCC state (to protect you from dodgy suppliers), so the data supplied from the DCC is the source of truth over the data that is supplied by the Home Mini.

Therefore in the app, they show the mini data for periods they don't have the "true" data for, and then replace it with the data they receive by the DCC when available. This is why in my example above there were discrepencies because the Mini reported one thing and the DCC reported another to OE.

For example, if you looked at the app for a couple of days ago and compare it to the current consumption data of the same timeframe, they will not necessarily match (as i showed in my example above).

So why then, when we get to midnight, doesn't it rollover the correct consumption for today, into yesterday? Instead, the data is updated by HomeMini around 7am with a figure that doesn't quite match what the SmartMeter reported throughout the day yesterday?

Based on above, the previous consumption data is soley fed from the data reported by your meter to OE (the source of truth), so that everyone (whether they have a mini or not) can see accurate bills. These are different APIs to the ones that supply data from the home mini.

The data is not updated by the Mini at 7am, as it doesn't come into the equation. The data is updated by the DCC/OE at 7am and is why it's not guaranteed to match the data from the previous day reported by the Home Mini.

As mentioned, I can't do what OE do on their app (show home mini data for the previous day until available via the DCC), because HA entities are not built to change historic data and the previous day isn't always the previous day for everyone.

This is also why in the docs, I state the entity represents data from your meter not meter or mini.

I guess it means I just can't rely on the data in the 'Previous Accumulative Consumption' sensors being accurate.

Nope the opposite. This previous consumption sensors are the source of truth to how you'll be billed. They are the most accurate. The current consumption entities are just to guide you in real time.

Based on this conversation, I'll elevate these explainations into the docs and FAQs.

[smithbill17]

Thank you Dave for taking the time to explain this & for proposing to update the FAQs & docs.

I'm now getting the picture somewhat - it seems to boil down to the fact that the HomeMini is not reporting the same data that the SmartMeter sends every 30mins to the DCC. That's a bit concerning, as who's to say the SmartMeter/DCC isn't wrong & the HomeMini might actually be right??

If I've understood correctly, and the SmartMeter/DCC data is assumed to be correct, and that data is reported by the meter to the DCC every 30mins, isn't it possible to retrieve that data into HomeAssistant every 30mins? I guess the answer is no, it's not available to retrieve from the DCC until about 7am the next day. And because HomeAssistant doesn't allow changing historic data, you couldn't do anything with it anyway.

So, wouldn't it be possible to create some kind of 'variance' sensor that updates when the DCC data arrives and records the discrepancy between HomeMini & DCC data? Perhaps that would be a pointless exercise.

And it still leaves the fundamental question for Octopus Energy as to why HomeMini data is different to DCC data.

[BottlecapDave]

The meter is always correct. The home mini gets the data from the meter, but the reporting of this sometimes is out. My assumption is this due to 1) the mini losing connection with the meter 2) the mini losing connection to OE API 3) some sort of rounding issue on the mini

As mentioned, this question would need to be directed to Octopus for the proper answer.

Only official hardware can connect to the meter (like a highglow), and would be beyond the scope of this integration. So we have to rely on datbebing available via the APIs

Why is the data not available straight away? Again this would be a question directed at OE. I just get data when it's available via the API.

I have no plans to add any variance sensor. You could accomplish this yourself with a utility meter to store the previous days worth of current accumulative consumption and then a template sensor to compare the utility meter value with the previous consumption sensor.