search

BottlecapDave / HomeAssistant-OctopusEnergy

Unofficial Home Assistant integration for interacting with Octopus Energy
https://bottlecapdave.github.io/HomeAssistant-OctopusEnergy/
MIT License
571 stars 57 forks source link

Previous Accumulative Sensors not Updating #780

Closed darrenmckie closed 6 months ago

darrenmckie commented 7 months ago

Describe the bug

The previous accumulative sensors for gas and electricity are not updating.

Reproduction steps

Look at the sensor values.

Here is my previous accumulative gas sensor that has not updated for 3 days:

state_class: total last_reset: "2024-02-24T00:00:00+00:00" mprn: "redacted" serial_number: redacted is_estimated: true charges:

Expected behaviour

The sensors should have the previous day values.

Tariff Code

G-1R-SILVER-23-12-06-M

Integration Version

10.1.4

Home Assistant Version

2024.2.2

Fresh Install?

After upgrading

Home Assistant Logs

"{ "home_assistant": { "installation_type": "Home Assistant Container", "version": "2024.2.2", "dev": false, "hassio": false, "virtualenv": false, "python_version": "3.12.1", "docker": true, "arch": "x86_64", "timezone": "Europe/London", "os_name": "Linux", "os_version": "5.13.x", "run_as_root": true }, "custom_components": { "waste_collection_schedule": { "version": "1.45.1", "requirements": [ "icalendar", "recurring_ical_events", "icalevents", "beautifulsoup4", "lxml" ] }, "octopus_energy": { "version": "10.1.4", "requirements": [] }, "dvla": { "version": "2023.9.0", "requirements": [] }, "hacs": { "version": "1.34.0", "requirements": [ "aiogithubapi>=22.10.1" ] }, "pirateweather": { "version": "1.4.2", "requirements": [ "python-forecastio==1.4.0" ] }, "asusrouter": { "version": "0.29.0", "requirements": [ "asusrouter==1.7.0" ] } }, "integration_manifest": { "domain": "octopus_energy", "name": "Octopus Energy", "codeowners": [ "@bottlecapdave" ], "config_flow": true, "dependencies": [ "repairs", "recorder" ], "documentation": "https://bottlecapdave.github.io/HomeAssistant-OctopusEnergy", "homekit": {}, "iot_class": "cloud_polling", "issue_tracker": "https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/issues", "ssdp": [], "version": "10.1.4", "zeroconf": [], "is_built_in": false }, "data": { "id": "REDACTED", "octoplus_enrolled": true, "electricity_meter_points": [ { "mpan": "REDACTED", "meters": [ { "serial_number": "REDACTED", "is_export": false, "is_smart_meter": true, "device_id": "REDACTED", "manufacturer": "1031 - Secure Meters UK", "model": "C0A10101", "firmware": "C0A10003" } ], "agreements": [ { "start": "2019-03-16T00:00:00+00:00", "end": "2019-05-22T23:00:00+00:00", "tariff_code": "E-1R-MSM-FIX-12M-19-02-19-M", "product_code": "MSM-FIX-12M-19-02-19" }, { "start": "2019-05-22T23:00:00+00:00", "end": "2020-05-22T23:00:00+00:00", "tariff_code": "E-1R-AGILE-18-02-21-M", "product_code": "AGILE-18-02-21" }, { "start": "2020-05-22T23:00:00+00:00", "end": "2021-05-22T23:00:00+00:00", "tariff_code": "E-1R-AGILE-18-02-21-M", "product_code": "AGILE-18-02-21" }, { "start": "2021-05-22T23:00:00+00:00", "end": "2021-09-22T23:00:00+00:00", "tariff_code": "E-1R-AGILE-18-02-21-M", "product_code": "AGILE-18-02-21" }, { "start": "2021-09-22T23:00:00+00:00", "end": "2022-09-22T23:00:00+00:00", "tariff_code": "E-1R-GO-21-09-23-M", "product_code": "GO-21-09-23" }, { "start": "2022-09-22T23:00:00+00:00", "end": "2023-03-31T23:00:00+00:00", "tariff_code": "E-1R-VAR-22-04-02-M", "product_code": "VAR-22-04-02" }, { "start": "2023-03-31T23:00:00+00:00", "end": "2023-10-07T23:00:00+00:00", "tariff_code": "E-1R-VAR-22-11-01-M", "product_code": "VAR-22-11-01" }, { "start": "2023-10-07T23:00:00+00:00", "end": "2024-02-15T00:00:00+00:00", "tariff_code": "E-1R-AGILE-FLEX-22-11-25-M", "product_code": "AGILE-FLEX-22-11-25" }, { "start": "2024-02-15T00:00:00+00:00", "end": "2025-02-15T00:00:00+00:00", "tariff_code": "E-1R-AGILE-23-12-06-M", "product_code": "AGILE-23-12-06" } ] } ], "gas_meter_points": [ { "mprn": "REDACTED", "meters": [ { "serial_number": "REDACTED", "consumption_units": "m\u00b3", "is_smart_meter": true, "device_id": "REDACTED", "manufacturer": "1031 - Secure Meters UK", "model": "C4A10501", "firmware": "C4A10003" } ], "agreements": [ { "start": "2019-03-16T00:00:00+00:00", "end": "2020-03-16T00:00:00+00:00", "tariff_code": "G-1R-MSM-FIX-12M-19-02-19-M", "product_code": "MSM-FIX-12M-19-02-19" }, { "start": "2020-03-16T00:00:00+00:00", "end": "2021-03-16T00:00:00+00:00", "tariff_code": "G-1R-FIX-12M-20-02-12-M", "product_code": "FIX-12M-20-02-12" }, { "start": "2021-03-16T00:00:00+00:00", "end": "2023-03-31T23:00:00+00:00", "tariff_code": "G-1R-VAR-20-10-01-M", "product_code": "VAR-20-10-01" }, { "start": "2023-03-31T23:00:00+00:00", "end": "2024-01-26T00:00:00+00:00", "tariff_code": "G-1R-VAR-22-11-01-M", "product_code": "VAR-22-11-01" }, { "start": "2024-01-26T00:00:00+00:00", "end": "2024-02-15T00:00:00+00:00", "tariff_code": "G-1R-SILVER-FLEX-22-11-25-M", "product_code": "SILVER-FLEX-22-11-25" }, { "start": "2024-02-15T00:00:00+00:00", "end": "2025-02-15T00:00:00+00:00", "tariff_code": "G-1R-SILVER-23-12-06-M", "product_code": "SILVER-23-12-06" } ] } ] } } "

Confirmation

red73ver commented 7 months ago

Hi, I lost octopus_energy_electricity_<number>_<number>_export_previous_accumulative_costsensor completely after recent upgrade to 10.0.4, and this problem still exist in current version. Is this a bug or this entity has been removed ?

BottlecapDave commented 7 months ago

@darrenmckie sorry you're seeing this issue. The data you posted states the data was last retrieved at 2024-02-27T19:54:03.760489, but last evaluated at 2024-02-25T18:49:31.849457, which implies that the previous consumption data isn't available in time. As per the faq and the docs, the previous consumption data always looks at the day before. Therefore if the data takes longer to refresh, it won't populate. If you check the site, is the data available for the previous day? If not, that will be your problem. You can change the offset in the account config (see docs) if your data is normally late to arrive.

BottlecapDave commented 7 months ago

@red73ver it's best to only reply to issues you're seeing and not jump on other issues with different problems, otherwise they could get lost in the noise and you won't be helped.

Nothing has changed in how export meters are determined. Is that the only export sensor that is missing? If not, theres a good chance the data coming back from OE isn't stating your export meter properly (which is what the integration uses). There is this discussion which explains in more detail. If it's the only sensor, do you have any errors in your logs? Have you tried reloading the integration?

psfshr commented 7 months ago

@darrenmckie I've got the same here, even with the last_evaluated timestamp down to the minute. Are you seeing your Electricity with the last readings from the 26th by any chance?

I'm wondering if OE have had some issues, as my Mini also got kicked off my account over the weekend, and my consumption data on the web portal also stops at the same time.

I did email OE yesterday to ask what's going on, but they're suspiciously quiet so far...

darrenmckie commented 7 months ago

@darrenmckie sorry you're seeing this issue. The data you posted states the data was last retrieved at 2024-02-27T19:54:03.760489, but last evaluated at 2024-02-25T18:49:31.849457, which implies that the previous consumption data isn't available in time. As per the faq and the docs, the previous consumption data always looks at the day before. Therefore if the data takes longer to refresh, it won't populate. If you check the site, is the data available for the previous day? If not, that will be your problem. You can change the offset in the account config (see docs) if your data is normally late to arrive.

@BottlecapDave All of my data is available in the OE app. I have a mini that also shows today's usage in the app.

psfshr commented 7 months ago

@darrenmckie What do you get if you manually query the API as at https://octopus.energy/dashboard/new/accounts/personal-details/api-access ? I get the same returned data as I get in HA from the integration.

Re the app, I did notice before that the app seems to get it's data from someplace different to either the API or the OE website. Right now for me, neither the API or the website are showing the full set of data, yet the app does (I also have a mini connected).

I've noted that the I'm getting some errors in HA for the integration related to updates taking over 10s, so I suspect OE's API access is possibly overloaded?

BottlecapDave commented 7 months ago

If you have a mini configured, the app will show mini data where meter information isn't available. Last time I checked, this is not the situation in the web which just shows the data straight from your meter. My gas has been spotty at best for instance in the web.

psfshr commented 7 months ago

@BottlecapDave May or may not be related to this... I did turn on debug logging earlier to try and see if I could see what was going on, and it looks like the integration was trying to do a refresh every 30s? Is that usual?

2024-02-28 14:40:00.822 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching xxx_xxxxx_previous_consumption_and_rates data in 0.000 seconds (success: True) 2024-02-28 14:40:00.822 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching xxx_xxxxx_previous_consumption_and_rates data in 0.000 seconds (success: True) 2024-02-28 14:40:30.410 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.wheel_of_fortune] Finished fetching A-8772418B_wheel_of_fortune_spins data in 0.000 seconds (success: True) 2024-02-28 14:40:30.421 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.saving_sessions] Finished fetching A-8772418B_saving_sessions data in 0.000 seconds (success: True) 2024-02-28 14:40:30.460 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.electricity_rates] Finished fetching ELECTRICITY_RATES_xxx_xxxxx data in 0.000 seconds (success: True) 2024-02-28 14:40:30.460 DEBUG (MainThread) [custom_components.octopus_energy.electricity.current_rate] Updating OctopusEnergyElectricityCurrentRate for 'xxx/xxxxx' 2024-02-28 14:40:30.464 DEBUG (MainThread) [custom_components.octopus_energy.electricity.previous_rate] Updating OctopusEnergyElectricityPreviousRate for 'xxx/xxxxx' 2024-02-28 14:40:30.465 DEBUG (MainThread) [custom_components.octopus_energy.electricity.next_rate] Updating OctopusEnergyElectricityNextRate for 'xxx/xxxxx' 2024-02-28 14:40:30.501 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.electricity_standing_charges] Finished fetching ELECTRICITY_STANDING_CHARGES_xxx_xxxxx data in 0.000 seconds (success: True) 2024-02-28 14:40:30.501 DEBUG (MainThread) [custom_components.octopus_energy.electricity.standing_charge] Updating OctopusEnergyElectricityCurrentStandingCharge 2024-02-28 14:40:30.562 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.gas_rates] Finished fetching GAS_RATES_xxx_xxxxx data in 0.000 seconds (success: True) 2024-02-28 14:40:30.562 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_rate] Updating OctopusEnergyGasCurrentRate for 'xxx/xxxxx' 2024-02-28 14:40:30.566 DEBUG (MainThread) [custom_components.octopus_energy.gas.previous_rate] Updating OctopusEnergyGasPreviousRate for 'xxx/xxxxx' 2024-02-28 14:40:30.567 DEBUG (MainThread) [custom_components.octopus_energy.gas.next_rate] Updating OctopusEnergyGasNextRate for 'xxx/xxxxx' 2024-02-28 14:40:30.581 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.gas_standing_charges] Finished fetching GAS_STANDING_CHARGES_xxx_xxxxx data in 0.000 seconds (success: True) 2024-02-28 14:40:30.581 DEBUG (MainThread) [custom_components.octopus_energy.gas.standing_charge] Updating OctopusEnergyGasCurrentStandingCharge 2024-02-28 14:40:30.822 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching xxx_xxxxx_previous_consumption_and_rates data in 0.000 seconds (success: True) 2024-02-28 14:40:30.823 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching xxx_xxxxx_previous_consumption_and_rates data in 0.000 seconds (success: True) 2024-02-28 14:41:00.824 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching xxx_xxxxx_previous_consumption_and_rates data in 0.000 seconds (success: True) 2024-02-28 14:41:00.825 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching xxx_xxxxx_previous_consumption_and_rates data in 0.000 seconds (success: True) 2024-02-28 14:41:30.410 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.wheel_of_fortune] Finished fetching A-8772418B_wheel_of_fortune_spins data in 0.000 seconds (success: True) 2024-02-28 14:41:30.421 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.saving_sessions] Finished fetching A-8772418B_saving_sessions data in 0.000 seconds (success: True) 2024-02-28 14:41:30.460 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.electricity_rates] Finished fetching ELECTRICITY_RATES_xxx_xxxxx data in 0.000 seconds (success: True) 2024-02-28 14:41:30.461 DEBUG (MainThread) [custom_components.octopus_energy.electricity.current_rate] Updating OctopusEnergyElectricityCurrentRate for 'xxx/xxxxx' 2024-02-28 14:41:30.464 DEBUG (MainThread) [custom_components.octopus_energy.electricity.previous_rate] Updating OctopusEnergyElectricityPreviousRate for 'xxx/xxxxx' 2024-02-28 14:41:30.465 DEBUG (MainThread) [custom_components.octopus_energy.electricity.next_rate] Updating OctopusEnergyElectricityNextRate for 'xxx/xxxxx' 2024-02-28 14:41:30.501 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.electricity_standing_charges] Finished fetching ELECTRICITY_STANDING_CHARGES_xxx_xxxxx data in 0.000 seconds (success: True) 2024-02-28 14:41:30.501 DEBUG (MainThread) [custom_components.octopus_energy.electricity.standing_charge] Updating OctopusEnergyElectricityCurrentStandingCharge 2024-02-28 14:41:30.563 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.gas_rates] Finished fetching GAS_RATES_xxx_xxxxx data in 0.000 seconds (success: True) 2024-02-28 14:41:30.563 DEBUG (MainThread) [custom_components.octopus_energy.gas.current_rate] Updating OctopusEnergyGasCurrentRate for 'xxx/xxxxx' 2024-02-28 14:41:30.567 DEBUG (MainThread) [custom_components.octopus_energy.gas.previous_rate] Updating OctopusEnergyGasPreviousRate for 'xxx/xxxxx' 2024-02-28 14:41:30.568 DEBUG (MainThread) [custom_components.octopus_energy.gas.next_rate] Updating OctopusEnergyGasNextRate for 'xxx/xxxxx' 2024-02-28 14:41:30.582 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.gas_standing_charges] Finished fetching GAS_STANDING_CHARGES_xxx_xxxxx data in 0.000 seconds (success: True) 2024-02-28 14:41:30.582 DEBUG (MainThread) [custom_components.octopus_energy.gas.standing_charge] Updating OctopusEnergyGasCurrentStandingCharge 2024-02-28 14:41:30.824 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching xxx_xxxxx_previous_consumption_and_rates data in 0.000 seconds (success: True) 2024-02-28 14:41:30.825 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching xxx_xxxxx_previous_consumption_and_rates data in 0.000 seconds (success: True) 2024-02-28 14:42:00.825 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching xxx_xxxxx_previous_consumption_and_rates data in 0.000 seconds (success: True) 2024-02-28 14:42:00.826 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching xxx_xxxxx_previous_consumption_and_rates data in 0.000 seconds (success: True)

Is it refreshing too often and hitting some kind of API limit?

psfshr commented 7 months ago

If you have a mini configured, the app will show mini data where meter information isn't available. Last time I checked, this is not the situation in the web which just shows the data straight from your meter. My gas has been spotty at best for instance in the web.

So when all was working ok, I did get both "current" (Mini) and "previous" data in both the app and OE web, it's just when it falls over that the web seems to hit the same issue as the API, yet the app seems to be unaffected.

BottlecapDave commented 7 months ago

The background job I set to run every 1 minute, but it keeps its own internal tracking on how often to refresh the underlying data. This is to account for failures where it adjust the refresh times (this is all explained in the faq). There's a separate job per meter, so you'll see an update reference for each meter.

Regarding the web, I have not seen mini data being present when looking back at previous days, but like I said this might have changed.

If you update the offset for previous consumption meters to be more than 1 day behind, my guess is you'll start seeing data again because the data is taking longer than 24 hours to populate.

JRascagneres commented 7 months ago

I've just started seeing this issue too. I assume its something wrong on the Octopus side.

Nabys commented 7 months ago

I have this issue as well. Mine was last updated on the 25th.

red73ver commented 7 months ago

@BottlecapDave , Sorry, based on name with "accumulative sensors" i thought it will be best place to ask. Yes, I reinstalled extension completely , and setup integration from scratch. Now I see both counters marked as "import" instead of pair of "import + export". Additionally, in the device (MPAN) which is the export one, now I see "Previous Accumulative Consumption" and "Previous Accumulative cost" ( with no export prefix ) as unknown. I think it might be an issue with the data coming from OE but I don't know how to check it. What I was able to get from "Device Diagnostic" is "is_export": false, in the JSON on both counters. Is there any way I can query what data I'm getting from OE to confirm where the problem exist ?

BottlecapDave commented 7 months ago

@red73ver As per the discussion, you're looking for the presence of smartExportElectricityMeter. This is what is used to determine if your meter is an export meter or not, with to None or null indicating it's not an export meter. With your logs set to debug (see FAQ) and the integration reloaded, you'll see a log with account: with the response from the query outlined at https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/blob/develop/custom_components/octopus_energy/api_client/__init__.py#L30. As per the discussion, they had to contact OE support.

BottlecapDave commented 7 months ago

If the data is not coming through and the data_last_retrieved value is updating regularly, it means that OE does not have the data for the time period the integration is requesting. By default, the sensors request data for the last day (e.g. today it will be requesting for 28/2/24). If the data for this date arrives in OE servers tomorrow, then the sensor will never update it. You can adjust the time that it looks back (e.g. always look two days back instead of one), but setting the offset in your integration config.

I plan to add a new sensor in a future release to indicate the datetime of the latest data available for your meter from the OE servers (e.g. if there is only data up to midnight of 25/2/24, then the sensor will show this time), to make this situation clearer as the website/app can be misleading especially if you have a device like a home mini.

CambsWolfie commented 7 months ago

Hello David,Thank you very much for all your efforts developing the software it’s brilliant and really appreciated.I’ve also noticed this problem over the last few days and have just rolled back to 10.1.0 from 10.1.4. The previous days costs now appear to be working correctly. Hope this helpsOn 29 Feb 2024, at 06:10, David Kendall @.***> wrote: If the data is not coming through and the data_last_retrieved value is updating regularly, it means that OE does not have the data for the time period the integration is requesting. By default, the sensors request data for the last day (e.g. today it will be requesting for 28/2/24). If the data for this date arrives in OE servers tomorrow, then the sensor will never update it. You can adjust the time that it looks back (e.g. always look two days back instead of one), but setting the offset in your integration config. I plan to add a new sensor in a future release to indicate the datetime of the latest data available for your meter from the OE servers (e.g. if there is only data up to midnight of 25/2/24, then the sensor will show this time), to make this situation clearer as the website/app can be misleading especially if you have a device like a home mini.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

edwardhorsford commented 7 months ago

I've also not had updated previous readings since the 25th.

last_reset: 2024-02-25T00:00:00+00:00
last_evaluated: 2024-02-26T10:55:28.058332+00:00
data_last_retrieved: 2024-02-29T09:35:50.454630+00:00
lildude commented 7 months ago

I'm experiencing this too. Nothing in HA since the 25th, but the consumption API is returning more recent data:

$ curl -su "$TOKEN" "https://api.octopus.energy/v1/electricity-meter-points/[REDACTED]/meters/[REDACTED]/consumption/?group_by=day" | jq ".results[:5]"
[
  {
    "consumption": 32.448,
    "interval_start": "2024-02-27T00:00:00Z",
    "interval_end": "2024-02-28T00:00:00Z"
  },
  {
    "consumption": 11.534,
    "interval_start": "2024-02-26T00:00:00Z",
    "interval_end": "2024-02-27T00:00:00Z"
  },
  {
    "consumption": 8.827,
    "interval_start": "2024-02-25T00:00:00Z",
    "interval_end": "2024-02-26T00:00:00Z"
  },
  {
    "consumption": 5.027,
    "interval_start": "2024-02-24T00:00:00Z",
    "interval_end": "2024-02-25T00:00:00Z"
  },
  {
    "consumption": 5.239,
    "interval_start": "2024-02-23T00:00:00Z",
    "interval_end": "2024-02-24T00:00:00Z"
  }
]
$
yiannisc commented 7 months ago

I am having the same issue.

`Mpan XXXXXXXXXXXX Serial number YYYYYYYYYY Is export false Is smart meter true Total 9.74 Charges

JRascagneres commented 7 months ago

Posting the same over and over isn't really helpful. As Dave has already stated it appears that from this data Octopus' API is not supplying the consumption data. There are reports all over the web of the same issues. There is an issue with Octopus not this integration. The 'Data last retrieved' shows that the data WAS pulled by the integration from the API.

stuartoram commented 7 months ago

The API does appear to be beginning to catch up, but for the accounts I have visibility of (4 total), these are still 48hrs vs 24hrs behind, so these sensors won't start working again until Octopus resolve this.

red73ver commented 7 months ago

When i do query curl -su "$API_KEY:" "https://api.octopus.energy/v1/electricity-meter-points/<number>/meters/<sn>/consumption/" im getting below {"count":0,"next":null,"previous":null,"results":[]}. I did ask OE support for checking this, along with checking why data from "https://api.octopus.energy/v1/graphql/ have no indication of smartExportElectricityMeter as mentioned by @BottlecapDave . We will see what they will come back with.

darrenmckie commented 6 months ago

The sensors do now work up to 48 hours ago. It is frustrating that it used to be 24 hours ago, but I suspect it is something at OE end. I will close this issue.

Thanks @BottlecapDave for everything you do with this integration.

darrenmckie commented 6 months ago

The sensors do now work up to 48 hours ago. It is frustrating that it used to be 24 hours ago, but I suspect it is something at OE end. I will close this issue.

Thanks @BottlecapDave for everything you do with this integration.