Export tracker issue

[alfwro13]

Describe the bug

I woke up this morning and have noticed that overnight I somehow exported nearly £800 worth of energy according to the energy tracker provided by this integration:

state_class: total
last_reset: 2024-04-08T00:00:00+01:00
name: export
mpan: 1900092354229
target_entity_id: sensor.shellyem_channel_1_energy_returned
entity_accumulative_value: true
weekday_reset: 0
month_day_reset: 1
kind: cost_tracker
is_tracking: true
tracked_charges: 
- start: '2024-06-08T00:00:00+01:00'
  end: '2024-06-08T00:30:00+01:00'
  rate: 0.15
  consumption: 0.007700000000113505
  cost: 0.0011550000000170257
- start: '2024-06-08T00:30:00+01:00'
  end: '2024-06-08T01:00:00+01:00'
  rate: 0.15
  consumption: 0.00029999999992469384
  cost: 0.00004499999998870407
- start: '2024-06-08T01:00:00+01:00'
  end: '2024-06-08T01:30:00+01:00'
  rate: 0.15
  **consumption: 1764.6992
  cost: 264.70488**
- start: '2024-06-08T01:30:00+01:00'
  end: '2024-06-08T02:00:00+01:00'
  rate: 0.15
  consumption: 0.00009999999997489795
  cost: 0.000014999999996234692
- start: '2024-06-08T02:00:00+01:00'
  end: '2024-06-08T02:30:00+01:00'
  rate: 0.15
  consumption: 0.0018000000000029104
  cost: 0.00027000000000043656
- start: '2024-06-08T02:30:00+01:00'
  end: '2024-06-08T03:00:00+01:00'
  rate: 0.15
  **consumption: 1764.7014
  cost: 264.70521**
- start: '2024-06-08T04:00:00+01:00'
  end: '2024-06-08T04:30:00+01:00'
  rate: 0.15
  consumption: 0.00009999999997489795
  cost: 0.000014999999996234692
- start: '2024-06-08T04:30:00+01:00'
  end: '2024-06-08T05:00:00+01:00'
  rate: 0.15
  **consumption: 1764.7023
  cost: 264.70534499999997**
- start: '2024-06-08T05:00:00+01:00'
  end: '2024-06-08T05:30:00+01:00'
  rate: 0.15
  consumption: 0.00009999999997489795
  cost: 0.000014999999996234692
- start: '2024-06-08T06:00:00+01:00'
  end: '2024-06-08T06:30:00+01:00'
  rate: 0.15
  consumption: 1.0064000000002125
  cost: 0.15096000000003187
- start: '2024-06-08T06:30:00+01:00'
  end: '2024-06-08T07:00:00+01:00'
  rate: 0.15
  consumption: 0.08599999999978536
  cost: 0.012899999999967804

untracked_charges: 
total_consumption: 5295.205400000001
unit_of_measurement: GBP
device_class: monetary
icon: mdi:currency-gbp
friendly_name: Octopus Energy Cost Tracker export

I also have a energy meter using the very same entity (sensor.shellyem_channel_1_energy_returned) and that is reporting the correct amount so I know the issue is not with data reported by that entity.

Reproduction steps

Setup export tracker and let it run

Expected behaviour

It should accurately track energy usage - same way utility meter does

Tariff Code

E-1R-AGILE-23-12-06-J

Integration Version

11.0.2

Home Assistant Version

2024.6.0

Fresh Install?

After upgrading

Home Assistant Logs

debug logging was not enabled at the time this has happened - it is running now so when that occurs I'll update

Confirmation

• [X] I confirm that I cannot find my solution within the documentation
• [X] I confirm that I cannot find my solution within the FAQ

[BottlecapDave]

Hello and sorry you're seeing this issue.

The cost tracker listens for state events raised by HA for the target entities. It then accumulates the difference (if tracking an accumulative sensor) or the provided value. This suggests that for some reason HA provided this integration with the wrong value. Now that you have debugging enabled, there will be logs in the form of event: which will hold the raw value that HA provided the integration.

You can use this service to manually adjust the effected time periods manually to correct this mistake.

I also have a energy meter using the very same entity (sensor.shellyem_channel_1_energy_returned) and that is reporting the correct amount so I know the issue is not with data reported by that entity.

I believe the energy meter looks at the raised statistics instead of state events, so are more staggered and less susceptable to fluxuations. Did the underlying sensor reset at all during these periods?

[alfwro13]

OK, it happened again - a 8:10 it was reporting correct value, 8:15 reported big jump:

I have extracted all the event: entries from the log file that refer to the export tracker:

2024-06-08 08:11:01.110 DEBUG (MainThread) [custom_components.octopus_energy.cost_tracker.cost_tracker] event: {'entity_id': 'sensor.shellyem_channel_1_energy_returned', 'old_state': <state sensor.shellyem_channel_1_energy_returned=1769.9398; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:06:11.734443+01:00>, 'new_state': <state sensor.shellyem_channel_1_energy_returned=1769.9399; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:11:01.108960+01:00>}
2024-06-08 08:12:01.001 DEBUG (MainThread) [custom_components.octopus_energy.cost_tracker.cost_tracker] event: {'entity_id': 'sensor.shellyem_channel_1_energy_returned', 'old_state': <state sensor.shellyem_channel_1_energy_returned=1769.9399; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:11:01.108960+01:00>, 'new_state': <state sensor.shellyem_channel_1_energy_returned=1769.9407; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:12:01.000606+01:00>}
2024-06-08 08:13:01.075 DEBUG (MainThread) [custom_components.octopus_energy.cost_tracker.cost_tracker] event: {'entity_id': 'sensor.shellyem_channel_1_energy_returned', 'old_state': <state sensor.shellyem_channel_1_energy_returned=1769.9407; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:12:01.000606+01:00>, 'new_state': <state sensor.shellyem_channel_1_energy_returned=1769.9411; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:13:01.074244+01:00>}
2024-06-08 08:14:01.025 DEBUG (MainThread) [custom_components.octopus_energy.cost_tracker.cost_tracker] event: {'entity_id': 'sensor.shellyem_channel_1_energy_returned', 'old_state': <state sensor.shellyem_channel_1_energy_returned=1769.9411; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:13:01.074244+01:00>, 'new_state': <state sensor.shellyem_channel_1_energy_returned=1769.9415; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:14:01.024270+01:00>}
2024-06-08 08:18:10.055 DEBUG (MainThread) [custom_components.octopus_energy.cost_tracker.cost_tracker] event: {'entity_id': 'sensor.shellyem_channel_1_energy_returned', 'old_state': <state sensor.shellyem_channel_1_energy_returned=1769.9417; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:15:01.017404+01:00>, 'new_state': <state sensor.shellyem_channel_1_energy_returned=1769.9416; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:18:10.047549+01:00>}

I don't mind posting the whole debug file but it is quite big and it has a lot of irrelevant data

[alfwro13]

- start: '2024-06-08T08:00:00+01:00'
end: '2024-06-08T08:30:00+01:00'
rate: 0.15
consumption: 1770.0171000000003
cost: 265.502565

The data recorded by the tracker (1770.0171000000003) strangely similar to the value reported by shelly EM: 1769.9407

[BottlecapDave]

hmm. I think the problem is this line

2024-06-08 08:18:10.055 DEBUG (MainThread) [custom_components.octopus_energy.cost_tracker.cost_tracker] event: {'entity_id': 'sensor.shellyem_channel_1_energy_returned', 'old_state': <state sensor.shellyem_channel_1_energy_returned=1769.9417; state_class=total_increasing, unit_of_measurement=kWh, device_class=energy, friendly_name=(Ch1) Grid Export @ 2024-06-08T08:15:01.017404+01:00>, 'new_state': <state sensor.shellyem_channel_1_energy_returned=1769.9416;

With this the value goes from 1769.9417 to 1769.9416, which is a decrease, however the sensor is configured as a total_increasing sensor. From the HA docs

Similar to total, with the restriction that the state represents a monotonically increasing positive total which periodically restarts counting from 0, e.g. a daily amount of consumed gas, weekly water consumption or lifetime energy consumption. Statistics of the accumulated growth of the sensor's value since it was first added is updated every 5 minutes. A decreasing value is interpreted as the start of a new meter cycle or the replacement of the meter.

I have taken this to mean that if the value decreases, it mean it has reset and the value is the new starting value (i.e. take the value as is).

I'll need to investigate how I should interpret the state in this scenario.

[BottlecapDave]

I've added a threshold as per a comment in a HA issue to the cost calculation to handle scenarios where the total_increasing sensors "misbehave" and decrease slightly without a reset, which should fix your issue.

This fix is included in v11.2.0.