search

BottlecapDave / HomeAssistant-OctopusEnergy

Unofficial Home Assistant integration for interacting with Octopus Energy
https://bottlecapdave.github.io/HomeAssistant-OctopusEnergy/
MIT License
612 stars 61 forks source link

Octopus Intelligent - Current Rate is out of sync with intelligent dispatching #732

Closed giuliodali closed 8 months ago

giuliodali commented 9 months ago

Describe the bug

I'm finding my electricity prices in HA are substantially off and I've tracked it down to the fact that when the intelligent rate is dispatching the current rate entity either hasn't updated at all, or the update is delayed and hence tracking at a higher price.

You can see from the below screenshot, with the dispatching events at the top and the rate below. The red arrows point to the times where the event hasn't updated the price accordingly and hence tracks a higher cost.

image

Reproduction steps

Other than being on an intelligent plan, I'm not sure how else to repro.

Expected behaviour

When intelligent starts or stops dispatching, the price is refreshed immediately to reflect what will be charged from that point onwards.

Tariff Code

E-1R-INTELLI-VAR-22-10-14-J

Integration Version

10.0.4

Home Assistant Version

2024.1.6

Fresh Install?

Not specified

Home Assistant Logs

?

Confirmation

RobinXe commented 9 months ago

I've found this happening if a dispatch starts as soon as I plug in, and I believe it might be due to a disconnect between the polling frequency of dispatches vs current rate?

I suspect that care should probably be taken in assuming a rate based solely on the dispatch being on, as I think OE bill at full price for on-peak dispatches if the car doesn't actually charge.

This might be rectifiable by having a change to the dispatch state trigger an unscheduled current rate update, or it might be best left to the user to implement by corroboration with other sensors, such as the Zappi integration's 'boosting' state, or comparable for other chargers/vehicles.

If the OE API doesn't indicate the off-peak price faithfully in conjunction with on-peak dispatches, then there's a danger in artificially lowering the price of the sensor, as users may have other load triggers hanging off the price sensor, which may start consuming at the higher rate.

Just my thoughts. I'd like a solution as well, but I'll probably end up adding my own sensor, which corroborates the charging status of the vehicle with other sensors, before lowering the price to next rate, if next rate < current rate, when dispatch is on.

BottlecapDave commented 9 months ago

Thanks for raising this. I've updated the background service previously to include the correct rates - this should be shown if you monitor the rate event sensors. However, the rate sensors themselves have been setup to only update every 30 minutes which I'll look at changing.

As for the chance of invalid data being provided, this is highlighted in the docs for the intelligent sensors, but not the rate sensors. I'll update these to include this disclaimer. There is an attribute on the rate sensors that indicate if the rate has been adjusted due to an intelligent dispatch outside of the standard off peak times, which can be checked and reacted accordingly.

giuliodali commented 9 months ago

I think updating the rate sensors every 30 minutes makes sense as a generic rule. In the case of a dispatching event, can it update at that point in time (outside of the regular 30 min interval) to get the latest rate? This would keep it in sync.

BottlecapDave commented 9 months ago

This is how it currently is, but the rate sensors only look at the updated information on the 30 minute mark. Therefore if dispatch information is updated at say 08:32, the rate sensors wouldn't typically update until 09:00. It's this logic that I will look at updating.

Mutley247365 commented 9 months ago

Ive also lost current_consumption and previous consumption as part of this too from my Flux tariff,

BottlecapDave commented 9 months ago

@Mutley247365 I'm not sure how this issue can be related to you losing entities. Please raise a separate ticket with more information.

giuliodali commented 9 months ago

I had a quick check this morning and new one happened. The rate went up in the middle of the nights dispatch window.

image

RobinXe commented 9 months ago

I think it's the same issue. The period indicated is before the 2330-0530 nightly off-peak, so you've had another on-peak dispatch, that seems to have correctly assessed the rate at the start of the period, but reverted until the start of the off-peak period.

Hopefully Dave's proposed revision to the price sensor logic, in relation to the intelligent dispatching, will catch this situation too.

mcc05 commented 9 months ago

Sorry to ask is this the same issue on the costing side, had slots allocated last yesterday 21:30 -22:00 ,then 22:30 onwards for car charging. Looked at the cost today and it didn't add up to low rate x kWh, so looked at the data, is this Octopus not updating what the actual consumption rates on the time periods? image

BottlecapDave commented 9 months ago

@mcc05 The costs are calculated from the completed dispatches returned by OE. Therefore if they don't provide a completed dispatch, then old costs will not be adjusted. Please look at the completed dispatches to see if they account for the mismatches. If they don't, please raise a separate issue with more information on your situation.

BottlecapDave commented 9 months ago

This issue should now be fixed in v10.1.0

RobinXe commented 9 months ago

Might still have an issue picking up no notice dispatches? Got a cheeky one when plugged the car back in at about 1118, til 1130. Might be some info in the fact that it triggered the dispatch 'on' twice in that period? Also, at the time the dispatch re-triggered, the current rate updated, but did not change.

image
BottlecapDave commented 9 months ago

I assume you've updated and restarted your HA instance. What does the event entity state for todays rates? I'm not sure how this would be happening as all data is re-evaluated every minute.

RobinXe commented 9 months ago

Yeah, updated and restarted. If it's just reevaluating then I'm guessing it wouldn't pick up a dispatch that wasn't there previously, without fetching new rates from the API?

Edit: I realise that's nonsense, as it picks up the dispatch fine.

BottlecapDave commented 9 months ago

Yeah the required data is there and should be stitched together when either the dispatch or rate information updates. The rate should then be triggered with the new data. I'm not on intelligent, so I'll run a simulation on my test device and try and work out what's happening. Might take a little bit of time to work out.

RobinXe commented 9 months ago

Very grateful to you mate! Let me know if I can offer any support/info

giuliodali commented 9 months ago

Firstly, many thanks for looking into this. I've upgraded to 10.1.0 earlier today (and restarted HA) and I've still got incorrect rates BUT I've spotted a scenario. The intelligent dispatch that I was given was split into 2 consecutive slots namely:

Planned dispatches

This resulted in this chart which looks correct at 21:30, but then broke at 22:00 when we're still dispatching but on the second consecutive slot.

image

Hopefully this helps.

BottlecapDave commented 9 months ago

Hello. This should hopefully be fixed in v10.1.1. Please confirm when you can :)

RobinXe commented 9 months ago

Doesn't appear to be, sorry Dave; had a cheeky couple of minutes when I plugged the car back in just before 1100 this morning, and the dispatch registered while the current price didn't.

image
BottlecapDave commented 9 months ago
  1. Could you please run the diagnostics in the FAQ and paste the results here.
  2. Could you please post a picture of the attributes for your current rate sensor?
  3. When you say plugged the car in, is Octopus coming up with the plan or are you bump charging?
BottlecapDave commented 9 months ago

Could you also please supply the planned dispatches outlined during this period in your dispatching entity?

giuliodali commented 9 months ago

Seems to be stuck on the hight rate for me. I updated and restarted during the ON session before 16:00.

Screenshot_20240217_231618~2

Planned dispatches

Completed dispatches

BottlecapDave commented 9 months ago

A couple of issues/notes from what you've posted have been discovered

  1. There is an bug in the consistency between how dispatches are used between the rates and the dispatching sensor. The rates are only adjusted for planned dispatches that have a source of "smart-charge". This is so bump charges are ignored as these are not at off peak rates. The dispatching sensor however is not looking at the source, so is turning on more frequenty. This goes against the documented design of this sensor. This has been fixed in v10.1.2.
  2. Judging by your planned dispatches, it looks like OE has stopped supplying the source, which as stated above, is used to determine if the pending dispatch should be considered off peak. I've reached out to OE to see if this is a bug or a change in behaviour of the API. I don't want to assume that a source of null means smart-charge and is therefore off peak.
giuliodali commented 9 months ago

FYI, All the above are octopus decided charges and car was plugged in all day. No bump charges.

BottlecapDave commented 9 months ago

Yeah I feel Null source will be considered smart charges, but want to double check before making the change as I don't want people charged incorrectly. If you do a bump charge, do you get a pending dispatch come through and does it have a source? Understand if you don't want to do a bump charge due to the monetary cost.

I've raised the issue that the dispatch sensor won't turn on at all where pending dispatch source is not defined in https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/issues/766

giuliodali commented 9 months ago

Just tried a bump charge from octopus app and I got this:

Planned dispatches

And the bump charge sensor was turned on:

image

giuliodali commented 9 months ago

Also - Having just turned bump off (and car stopped charging), the bump & dispatch sensors are still on. I'm guessing they won't update until the next half hour? Is there an event that can detect the end of a bump charge?

giuliodali commented 9 months ago

Ok, now I have a normal charge coming through as this .

Planned dispatches

So it does look like the source is either null or smart-charge.

RobinXe commented 9 months ago
  1. Could you please run the diagnostics in the FAQ and paste the results here.
  2. Could you please post a picture of the attributes for your current rate sensor?
  3. When you say plugged the car in, is Octopus coming up with the plan or are you bump charging?

Do you still require this data Dave, or have you identified the source of the issue?

In answer to point 3, I plugged my car in just before 1100, with a ready time of 1100, so I got a snap Intelligent dispatch from about 1052 til 1100. It wasn't a bump charge.

BottlecapDave commented 9 months ago

Also - Having just turned bump off (and car stopped charging), the bump & dispatch sensors are still on.

@giuliodali If you toggle settings in the app, as per the FAQ, it can take up to 5 minutes for the data to refresh and update the sensors. If you toggle in the integration, it can again take up to 5 minutes for the dispatches to be refreshed. This is to try and account for rate limiting on the API. As mentioned, if you're not in v10.1.2, the dispatching sensor will come on during bump charges incorrectly.

So it does look like the source is either null or smart-charge. I'll await for confirmation from OE, but this does look like it's the case. Like I said, the last thing I want to do is turn it on during a bump charge and cost people money.

@RobinXe I think I've identified a few issues. The main issue you were seeing was that the price was calculated with one criteria and the dispatching sensor was calculated with another. They are now aligned, but both are sometimes not reporting correctly if OE doesn't provide an expected source (waiting for clarification)

RobinXe commented 9 months ago

Roger, thanks Dave!

Hopefully OE will sort the API out, I'm sure it's just an errant commit as they continually improve it. It would be good if they had some sort of channel to notify developers of changes, with an avenue for bug reporting. I'm unsure of how big that community would be though, beyond yourself!

In the absence of a differentiating feature, my personal preference would be for all dispatches to set the price low, since I don't bump charge, but I appreciate that everyone's use-case is different.

RobinXe commented 9 months ago

Morning, I got another no-notice dispatch when I plugged back in this morning and, looking at the planned dispatches in the integration, it appears that the source has been properly reported (if I've understood the issue correctly). The dispatch sensor turned on after a couple of minutes, but the current rate remained high. Hopefully this offers some insight, rather than confusion!

Screenshot_20240219_090447_Home Assistant

BottlecapDave commented 9 months ago

Could you please provide the diagnostics requested above which might provide some more insight. Could you also please provide the attributes of the current rate sensor that is not updating

RobinXe commented 9 months ago

Here you go!

IO_rate


  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2024.2.2",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.12.1",
    "docker": true,
    "arch": "x86_64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "6.1.74-haos",
    "supervisor": "2024.01.1",
    "host_os": "Home Assistant OS 11.5",
    "docker_version": "24.0.7",
    "chassis": "vm",
    "run_as_root": true
  },
  "custom_components": {
    "hon": {
      "version": "0.11.0-beta.1",
      "requirements": [
        "pyhOn==0.15.15"
      ]
    },
    "myenergi": {
      "version": "0.0.24",
      "requirements": [
        "pymyenergi==0.0.29"
      ]
    },
    "pyscript": {
      "version": "1.5.0",
      "requirements": [
        "croniter==1.3.8",
        "watchdog==2.3.1"
      ]
    },
    "ltss": {
      "version": "2.1.0",
      "requirements": [
        "sqlalchemy>=2.0,<3.0",
        "psycopg2-binary>=2.8,<3.0",
        "geoalchemy2>=0.13,<1.0"
      ]
    },
    "hacs": {
      "version": "1.34.0",
      "requirements": [
        "aiogithubapi>=22.10.1"
      ]
    },
    "solcast_solar": {
      "version": "3.0.47",
      "requirements": [
        "aiohttp>=3.6.2",
        "datetime>=4.3",
        "isodate>=0.6.0"
      ]
    },
    "nodered": {
      "version": "3.1.3",
      "requirements": []
    },
    "octopus_energy": {
      "version": "10.1.2",
      "requirements": []
    },
    "solaredge_modbus_multi": {
      "version": "2.4.11",
      "requirements": [
        "pymodbus>=3.5.4"
      ]
    }
  },
  "integration_manifest": {
    "domain": "octopus_energy",
    "name": "Octopus Energy",
    "codeowners": [
      "@bottlecapdave"
    ],
    "config_flow": true,
    "dependencies": [
      "repairs",
      "recorder"
    ],
    "documentation": "https://bottlecapdave.github.io/HomeAssistant-OctopusEnergy",
    "homekit": {},
    "iot_class": "cloud_polling",
    "issue_tracker": "https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/issues",
    "ssdp": [],
    "version": "10.1.2",
    "zeroconf": [],
    "is_built_in": false
  },
  "data": {
    "id": "A-87D88CCB",
    "octoplus_enrolled": true,
    "electricity_meter_points": [
      {
        "mpan": "**REDACTED**",
        "meters": [
          {
            "serial_number": "**REDACTED**",
            "is_export": true,
            "is_smart_meter": true,
            "device_id": null,
            "manufacturer": "1063 - Landis and Gyr",
            "model": "00050204",
            "firmware": "38040404"
          }
        ],
        "agreements": [
          {
            "start": "2023-08-12T23:00:00+00:00",
            "end": "2023-12-08T00:00:00+00:00",
            "tariff_code": "E-1R-OUTGOING-FIX-12M-19-05-13-H",
            "product_code": "OUTGOING-FIX-12M-19-05-13"
          },
          {
            "start": "2023-12-08T00:00:00+00:00",
            "end": "2024-01-22T00:00:00+00:00",
            "tariff_code": "E-1R-OUTGOING-LITE-FIX-12M-23-09-12-H",
            "product_code": "OUTGOING-LITE-FIX-12M-23-09-12"
          },
          {
            "start": "2024-01-22T00:00:00+00:00",
            "end": "2025-01-22T00:00:00+00:00",
            "tariff_code": "E-1R-OUTGOING-FIX-12M-19-05-13-H",
            "product_code": "OUTGOING-FIX-12M-19-05-13"
          }
        ]
      },
      {
        "mpan": "**REDACTED**",
        "meters": [
          {
            "serial_number": "**REDACTED**",
            "is_export": false,
            "is_smart_meter": true,
            "device_id": "**REDACTED**",
            "manufacturer": "1063 - Landis and Gyr",
            "model": "00050204",
            "firmware": "38040404"
          }
        ],
        "agreements": [
          {
            "start": "2022-11-16T00:00:00+00:00",
            "end": "2023-01-07T00:00:00+00:00",
            "tariff_code": "E-1R-VAR-22-10-01-H",
            "product_code": "VAR-22-10-01"
          },
          {
            "start": "2023-01-07T00:00:00+00:00",
            "end": "2023-02-02T00:00:00+00:00",
            "tariff_code": "E-1R-AGILE-FLEX-22-11-25-H",
            "product_code": "AGILE-FLEX-22-11-25"
          },
          {
            "start": "2023-02-02T00:00:00+00:00",
            "end": "2023-12-11T00:00:00+00:00",
            "tariff_code": "E-1R-SILVER-FLEX-22-11-25-H",
            "product_code": "SILVER-FLEX-22-11-25"
          },
          {
            "start": "2023-12-11T00:00:00+00:00",
            "end": "2024-01-22T00:00:00+00:00",
            "tariff_code": "E-1R-GO-VAR-22-10-14-H",
            "product_code": "GO-VAR-22-10-14"
          },
          {
            "start": "2024-01-22T00:00:00+00:00",
            "end": null,
            "tariff_code": "E-1R-INTELLI-VAR-22-10-14-H",
            "product_code": "INTELLI-VAR-22-10-14"
          }
        ]
      }
    ],
    "gas_meter_points": [
      {
        "mprn": "**REDACTED**",
        "meters": [
          {
            "serial_number": "**REDACTED**",
            "consumption_units": "m\u00b3",
            "is_smart_meter": true,
            "device_id": "**REDACTED**",
            "manufacturer": "1063 - Landis and Gyr",
            "model": "47720101",
            "firmware": "03035524"
          }
        ],
        "agreements": [
          {
            "start": "2022-11-16T00:00:00+00:00",
            "end": "2023-01-16T00:00:00+00:00",
            "tariff_code": "G-1R-VAR-22-10-01-H",
            "product_code": "VAR-22-10-01"
          },
          {
            "start": "2023-01-16T00:00:00+00:00",
            "end": "2023-02-02T00:00:00+00:00",
            "tariff_code": "G-1R-SILVER-FLEX-22-11-25-H",
            "product_code": "SILVER-FLEX-22-11-25"
          },
          {
            "start": "2023-02-02T00:00:00+00:00",
            "end": "2024-02-15T00:00:00+00:00",
            "tariff_code": "G-1R-SILVER-FLEX-22-11-25-H",
            "product_code": "SILVER-FLEX-22-11-25"
          },
          {
            "start": "2024-02-15T00:00:00+00:00",
            "end": "2025-02-15T00:00:00+00:00",
            "tariff_code": "G-1R-SILVER-23-12-06-H",
            "product_code": "SILVER-23-12-06"
          }
        ]
      }
    ]
  }
}```
BottlecapDave commented 9 months ago

Was that screenshot taken just now? I'd expect the last evaluated attribute to be updated roughly every minute. Do you have any errors in your logs?

RobinXe commented 9 months ago

I do indeed!

This error originated from a custom integration.

Logger: homeassistant
Source: custom_components/octopus_energy/utils/rate_information.py:60
Integration: Octopus Energy (documentation, issues)
First occurred: 08:42:39 (129 occurrences)
Last logged: 10:50:39

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 256, in _handle_refresh_interval
    await self._async_refresh(log_failures=True, scheduled=True)
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 412, in _async_refresh
    self.async_update_listeners()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 183, in async_update_listeners
    update_callback()
  File "/config/custom_components/octopus_energy/electricity/current_rate.py", line 104, in _handle_coordinator_update
    rate_information = get_current_rate_information(rates_result.rates, current)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/octopus_energy/utils/rate_information.py", line 60, in get_current_rate_information
    "tariff_code": current_rate["tariff_code"],
                   ~~~~~~~~~~~~^^^^^^^^^^^^^^^
KeyError: 'tariff_code'
RobinXe commented 9 months ago

After the end of the dispatch period, it is evaluating every minute.

BottlecapDave commented 9 months ago

Looks like another bug that I managed to introduce with some other fixes 🤦 . I'll try and get another fix out as soon as possible.

RobinXe commented 9 months ago

That's very much appreciated, thanks Dave! With the welcome pace and responsiveness you bring to improving this integration, the odd one is bound to slip through.

RobinXe commented 9 months ago

Can confirm, after this morning's dispatch-on-plugin, that the error only seems to occur during dispatches outside of the core off-peak hours, if that's of any assistance.

BottlecapDave commented 8 months ago

This and the original issue should be fixed in v10.1.3

RobinXe commented 8 months ago

Legend. Thanks for fitting this fix in during the working week Dave; I'll have another non-core-hours dispatch tomorrow morning (traffic permitting) and will provide confirmation.

giuliodali commented 8 months ago

Awesome, thank you!! :-) I updated last night and plugged in car and everything worked as expected. I even got a 30 minute slot this morning and it worked as expected. Much appreciated.

RobinXe commented 8 months ago

Can also confirm that dispatch and current rate sensors followed each other outside of core off-peak this morning, and no unusual errors were thrown. Nice work!