Gas current consumption and current consumption cost incorrect

[adam-cobb]

Describe the bug

Since the new price cap kicked in on the 1st January my _sensor.octopus_energy_gas_xxx_xxx_currentconsumption and _sensor.octopus_energy_gas_xxx_xxx_current_consumptioncost sensors are incorrect.

They reset at midnight as always, but then seem to immediately (within a couple of minutes) jump to a level of around 60kwh (and equivilent cost), and this starts as the base for the day effectively, so throwing all my figures and calculations based off of this.

Is this just an issue (seems suprising it's not been reported elsewhere by now if so?) or has something changed that I need to account for that I've missed?

Thanks.

Reproduction steps

Nothing has changed my side, figures just suddenly appear to be resetting then jumping to a value within a minute or two every day.

Expected behaviour

Consumption and cost for gas reset at midnight every day and gradually increase through usage as normal.

Tariff Code

G-1R-VAR-22-11-01-B

Integration Version

v9.2.1

Home Assistant Version

2023.12.1

Fresh Install?

Not specified

Home Assistant Logs

Can provide logs if it helps at all

Confirmation

• [X] I confirm that I cannot find my solution within the documentation
• [X] I confirm that I cannot find my solution within the FAQ

[adam-cobb]

I've just realised that these are Utility Meter helpers created that take data from the sensor.octopus_energy_gas_xxx_xxx_current_consumption and sensor.octopus_energy_gas_xxx_xxx_current_consumption_cost sensors.

So some confusion there, apologies. But still confused as these utility helpers worked fine prior to January 1st.

[adam-cobb]

Also, the electricity ones are setup exactly the same and they still work fine...

[BottlecapDave]

Hello. I haven't seen this behaviour myself. Can you submit what the value of the 'charges' attribute on the cost/consumption sensor looks like? Also what does the data look like in the octopus app for the current day in the daily breakdown? The sensor just reports back the results from the API and isn't doing anything special.

[adam-cobb]

@BottlecapDave sorry I can't seem to find the 'charges' attribute, I only see the following attributes on the sensor.octopus_energy_gas_xxx_xxx_current_consumption sensor

state_class: total last_reset: "2024-01-03T14:41:32.351590+00:00" unit_of_measurement: kWh device_class: energy icon: mdi:counter friendly_name: Current Gas Consumption last_evaluated: "2024-01-03T14:41:32.351590+00:00" data_last_retrieved: "2024-01-03T14:40:31.349774+00:00"

The data on Octopus is correct, as is the main energy dashboard in HA, just using the above sensors. The problem is my utility meter helper that is built on these sensors.

I've tried running the utility_meter.calibrate service against the cost and consumption helpers, resetting them to match the energy dashboard/Octopus for today. Hopefully it'll reset correctly tomorrow, but not sure why it would be any different.

[BottlecapDave]

The charges attribute is present on the current accumulative cost sensor. However I suspect this will look normal. I've had sporadic reports around the consumption sensor reporting weird spikes like this. However the fact your costs sensor is spiking too this suggests the data coming back from the API is wrong for a brief period, for which there's not a lot I can do. I'll try and add some additional logging for a future release to confirm this. Can you post the history view of the cost sensor itself?

[adam-cobb]

Unfortunately did it again last night :( Does seem like an issue with the utility helper rather than this integration, but confused as to why it started happening after the price change...

[BottlecapDave]

The utility meter is known to not play well with some of the sensors as they update whenever any of the underlying attributes or state update instead of just when the state updates. Therefore if the state doesn't change between updates (which are every minute), the ultity meter might add the current state again. However I wouldn't expect such a large jump at the start of the night as you're experiencing if this was the case.

People have gotten around this issue by creating a template sensor which just exposes the state without the attributes to confine the updates to just state updates.

What does the history of the underlying sensors look like?

[adam-cobb]

This is the last few days for the underlying consumption sensor (cost is much the same), seems odd as it has this big jump at midnight then goes negative the same amount, as if trying to correct it, but obviously that doesn't help once the utility meter has picked up that large positive spike -

However this is what it looked like in December (more normal) -

[BottlecapDave]

This definitely implies some dodgy data coming from the API. Could you zoom in to just the surrounding 10 minutes for that period of the high spike? Additional logging has been added when in debug mode which would also be useful for debugging (in the latest release)

[adam-cobb]

This is the 10 minute period around midnight last night -

I have updated to the latest version, but i'll need to turn on debug mode and capture the logs from tonight

[adam-cobb]

Here's two snippits from the debug log, at 12:00:34 last night when it first retreived the gas consumption data, you can see if retrieved a value of 79.34kWh and then 7 minutes later again with a negative of that first value -

Also to add, this happened for my electricicity consumption and cost as well last night for the first time, same spike and then negative correction value :(

2024-01-10 00:00:33.248 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.gas_rates] Finished fetching GAS_RATES_9381700301_E6S10962462061 data in 0.000 seconds (success: True)
2024-01-10 00:00:33.248 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_rate] Updating OctopusEnergyGasCurrentRate for '9381700301/E6S10962462061'
2024-01-10 00:00:33.250 DEBUG (MainThread) [custom_components.octopus_energy.gas.previous_rate] Updating OctopusEnergyGasPreviousRate for '9381700301/E6S10962462061'
2024-01-10 00:00:33.251 DEBUG (MainThread) [custom_components.octopus_energy.gas.next_rate] Updating OctopusEnergyGasNextRate for '9381700301/E6S10962462061'
2024-01-10 00:00:34.418 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.current_consumption] Finished fetching current_consumption_30-EB-5A-FF-FF-32-CB-5B data in 0.000 seconds (success: True)
2024-01-10 00:00:34.418 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_consumption] Updating OctopusEnergyCurrentGasConsumption
2024-01-10 00:00:34.418 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_consumption] state: 79.34110000000001; total_consumption: 79.34110000000001; previous_total_consumption: 79.34110000000001; consumption_data: [{'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 0, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 0, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 1, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 1, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 1, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 1, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 2, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 2, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 2, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 2, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 3, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 3, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 3, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 3, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 4, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 4, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 4, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 4, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 5, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 5, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 5, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 5, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 6, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 6, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 6, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 6, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 7, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 12.69054, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 7, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 7, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 7.22609, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 7, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 8, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 5.7113000000000005, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 8, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 8, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 5.31858, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 8, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 9, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 5.0043999999999995, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 9, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 9, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 3.19788, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 9, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 10, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 2.6929499999999997, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 10, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 10, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 10, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 11, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 11, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 11, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 11, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 12, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 12, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 12, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 12, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 13, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 2.91737, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 13, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 13, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 3.36619, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 13, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 14, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.22441, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 14, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 14, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 14, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 15, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 5.2737, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 15, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 15, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 15, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 16, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 3.88234, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 16, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 16, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 2.10948, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 16, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 17, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 5.0492799999999995, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 17, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 17, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 3.24276, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 17, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 18, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 4.80244, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 18, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 18, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 2.4460900000000003, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 18, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 19, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 19, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 19, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.47126999999999997, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 19, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 20, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 1.30159, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 20, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 20, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 2.41244, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 20, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 21, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 21, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 21, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 21, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 22, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 22, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 22, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 22, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 23, 0, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 23, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 9, 23, 30, tzinfo=datetime.timezone.utc)}, {'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 9, 23, 30, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 10, 0, 0, tzinfo=datetime.timezone.utc)}]
2024-01-10 00:00:34.419 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_accumulative_consumption_kwh] Calculated previous gas consumption for '9381700301/E6S10962462061'...
2024-01-10 00:00:34.420 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_accumulative_consumption_cubic_meters] Calculated previous gas consumption for '9381700301/E6S10962462061'...
2024-01-10 00:00:34.420 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_accumulative_cost] Calculated current gas consumption cost for '9381700301/E6S10962462061'...

2024-01-10 00:07:41.965 DEBUG (MainThread) [custom_components.octopus_energy.coordinators] last_retrieved: 2024-01-10 00:07:41.418437+00:00; request_attempts: 1; refresh_rate_in_minutes: 2; next_refresh: 2024-01-10 00:09:41.418437+00:00
2024-01-10 00:07:41.965 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.current_consumption] Finished fetching current_consumption_30-EB-5A-FF-FF-32-CB-5B data in 0.547 seconds (success: True)
2024-01-10 00:07:41.965 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_consumption] Updating OctopusEnergyCurrentGasConsumption
2024-01-10 00:07:41.965 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_consumption] state: -79.34110000000001; total_consumption: 0.0; previous_total_consumption: 0.0; consumption_data: [{'consumption': 0.0, 'demand': None, 'start': datetime.datetime(2024, 1, 10, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2024, 1, 10, 0, 30, tzinfo=datetime.timezone.utc)}]
2024-01-10 00:07:41.966 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_accumulative_consumption_kwh] Calculated previous gas consumption for '9381700301/E6S10962462061'...
2024-01-10 00:07:41.966 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_accumulative_consumption_cubic_meters] Calculated previous gas consumption for '9381700301/E6S10962462061'...
2024-01-10 00:07:41.966 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_accumulative_cost] Calculated current gas consumption cost for '9381700301/E6S10962462061'...
2024-01-10 00:07:48.239 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.gas_standing_charges] Finished fetching GAS_STANDING_CHARGES_9381700301_E6S10962462061 data in 0.000 seconds (success: True)
2024-01-10 00:07:48.239 DEBUG (MainThread) [custom_components.octopus_energy.gas.standing_charge] Updating OctopusEnergyGasCurrentStandingCharge
2024-01-10 00:07:49.247 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.wheel_of_fortune] Finished fetching A-EE60E952_wheel_of_fortune_spins data in 0.000 seconds (success: True)
2024-01-10 00:07:54.043 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching 1170001267622_19L2954590_previous_consumption_and_rates data in 0.000 seconds (success: True)
2024-01-10 00:07:54.044 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching 9381700301_E6S10962462061_previous_consumption_and_rates data in 0.000 seconds (success: True)
2024-01-10 00:07:56.300 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.account] Finished fetching update_account data in 0.000 seconds (success: True)

[BottlecapDave]

Hello. The issue with the current consumption sensor spikes should be fixed in v10.0.3.