Energy graph: Previous accumulative consumption appears as a single hourly figure and differs from Octopus dashboard

[gcoan]

Describe the bug

I have been using the GivEnergy inverter import entity and a 3-rate utility meter to track my consumption & display on the Energy dashboard. The figures have always reconciled well to the Octopus tariff usage for my account

Moved to agile and following the Energy Dashboard instructions to display consumption and costs.

But (1) the consumption entity doesn't agree with the Octopus portal, and (2) there's no hourly breakdown of cost, it all appears as a single spike.

e.g. 24/10. GivEnergy import=5.4+10.8=16.2kWh Octopus portal=16.34kWh - which is close enough previous_accumulation_consumption=16.71kWh

(blue are the GivEnergy utility meters)

And the previous accumulation consumption appears as a single spike, not spread hourly across the day (as the lighter blue givenergy registers are)

25th similar issue except Octopus entity much lower: GivEnergy meters=0.2+9.8+15.4=25.4kWh Octopus portal=25.43kWh Previous consumption entity=16.34kWh

26th Octopus much higher: GivEnergy meters=0.5+6+12.5=19kWh Octopus portal=19.1kWh Previous consumption entity=25.44kWh

Reproduction steps

added grid consumption to HA Energy dashboard following the instructions

Consumed energy=xxxx Previous Accumulated Consumption

Use an entity tracking the total costs=xxxxx Previous Accumulated Cost

Expected behaviour

Expect the energy graph to show consumption per hour over the day with a total consumption figure that's the same as the Octopus portal (and similar to that recorded from my inverters)

I suspect that the inconsistency in the previous accumulated kWh figures may be due to timing cut-offs of the meter data reaching Octopus and the API refreshing each night. If this is the case and its a common problem then this way of getting consumption from Octopus for the Energy dashboard is never going to work consistently and I'd be better using my existing inverter sensor and creating a template sensor (or finding one in predbat) that gives the correct agile rate per hour.

The data per hour in the graph I see you were trying to remove accumulation spikes, but the last post on the forum made me think it was solved by using external statistics. The previous_accumulation_sensor appears to have the data correctly structured to be displayed hourly:

state_class: total last_reset: 2023-10-26T23:00:00+00:00 mpan: xxxx serial_number: xxxx is_export: false is_smart_meter: true total: 25.914000000000005 last_calculated_timestamp: 2023-10-27T23:00:00+00:00 charges:

• from: '2023-10-26T23:00:00+00:00' to: '2023-10-26T23:30:00+00:00' consumption: 2.005
• from: '2023-10-26T23:30:00+00:00' to: '2023-10-27T00:00:00+00:00' consumption: 3.343
• from: '2023-10-27T00:00:00+00:00' to: '2023-10-27T00:30:00+00:00' consumption: 1.764

Tariff Code

E-1R-AGILE-FLEX-22-11-25-A

Integration Version

8.4.4

Home Assistant Version

2023.10.5

Fresh Install?

Not specified

Home Assistant Logs

-none-

[BottlecapDave]

Hello. Based on your spike, it looks like you might have configured the wrong entities for the energy dashboard. Unfortunately, HA doesn't support setting data historically as this causes issues with the reporting. Therefore, as per the docs, you need to use a separate sensor (Please note this starts with octopus_energy:, instead of normal sensor..

This will probably also be the reason why the data doesn't seem to align up, as with the normal entity, the accumulative data will be marked for one day, but will actually be for a previous day (or more if you've changed the default offset). For example, data that will be logged as for 29th October, will actually be for the 28th, but will appear in the dashboard on the 29th. As stated, the HA sensors are not really built for adding/altering historic data (i'm always keeping an eye on this).

If you update the energy dashboard, does this fix your issue?

[gcoan]

Thanks for your reply, sorry was a bit busy yesterday so couldn't respond. I thought I had configured it as per the docs and have had the Octopus integration running all the time so have built up history.

But now I look at the docs again I can see I was using the sensor.....previous_accumulative not octopus_energy...previous_accumulative for both the usage and the costs.

Have changed it to use the octopus_energy values and will leave it running, this looks like it may well cure the issue :-)

[BottlecapDave]

Yeah its a pain. With each HA release I try and see if there's an option that would allow the data to be associated with the normal sensor, as this would be a much better and expected user experience. This should fix your issue, but let me know if not.

[gcoan]

Sorry taken a while to prove that I had fixed it. Initially I just changed the dashboard to use the octopus_energy: but after a couple of days it was still showing the same daily spikes so I deleted that dashboard entry completely, left the energy dashboard to run through a full day cycle (clear the drains out), and then I tried again to add it. Was a bit difficult to find as there appeared to be several sensors that were near-identically named

But setting it up from scratch and adding the accumulated costs, it works perfectly, showing the different import prices and amounts throughout the day on all previous days:

You can see there's actually two sets of dark bluey grey import bar graph entries, that's because I was further able to extend the dashboard to show TODAY's electricity import as well as yesterday's by using the import sensors from my inverter:

Ignore that there are two entries for yesterday's consumption, now I have proved that the mechanism I have with Grid Import Today works, I will be removing the Previous Accumulative Consumption.
The Grid Import Today peak/day/offpeak are there from when I was on Flux and are consumption split by the Flux time periods - these will be removed as well, leaving just Grid Import today showing the day's consumption & correct agile pricing.

Do need to work out how to do an over-ride for the Octopus Power Up events which is what the Grid Import Today free is left over from. I have some different ideas as to how to do this.

I'll propose some changes and submit a pull request to the Octopus readme to explain how to do what I did. Thanks

[gcoan]

I created a PR for proposed documentation changes r/e the above, but the test pack failed with a whole load of errors on integration testing different tariff types even though I've changed the docs and not the code !

https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/actions/runs/6790919022/job/18461390954?pr=521

e.g. -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================== short test summary info ============================ FAILED tests/integration/api_client/test_async_get_octoplus_points.py::test_when_get_octoplus_points_is_called_then_points_are_returned - custom_components.octopus_energy.api_client.RequestError: Errors in request (https://api.octopus.energy/v1/graphql/): [{'message': 'Invalid data.', 'locations': [{'line': 2, 'column': 2}], 'path': ['obtainKrakenToken'], 'extensions': {'errorType': 'VALIDATION', 'errorCode': 'KT-CT-1137', 'errorDescription': 'Please provide the right combination of input fields.', 'errorClass': 'VALIDATION', 'validationErrors': [{'message': 'Please provide the right combination of input fields.', 'inputPath': ['obtainJsonWebTokenInput']}]}}]

[BottlecapDave]

Closing as I believe this issue has now been resolved. Please respond if not and we can reopen.

[gcoan]

all resolved thanks