Time period for Octopus data

[dunxd]

I have been using this HACS integration for a month or two, and it is great. I did some playing around with the Octopus API in Postman, and noticed that it gives consumption for half hourly time periods, with the group_by parameter allowing reduced granularity. The integration stores the data for the previous day. I understand that Octopus doesn't provide data newer than the previous day.

Do you have any plans to allow using a shorter time period in the integration? I'd be interested in helping out with this.

[BottlecapDave]

Hello. Because the data that Octopus Energy provide is old, I thought the previous day was the most useful. What sort of time periods were you thinking of?

[dunxd]

Ideally the smallest unit of time that HA can handle - that seems to be an hour as far as I can tell from the documentation I read. It would be helpful to see usage patterns through the previous days, and it may be possible to spot any anomalies or even set up alerts based on pattern changes. How complex is it to make this change?

[BottlecapDave]

I'm still not quite sure how this would work. HA sensors records the timestamp that states are changed and there doesn't seem to be any way to override this. This coupled with Octopus only providing historic data, would mean that the sensor data breakdown wouldn't match the time its updated. Could you give an example of the data that you would expect to be shown?

[dunxd]

I would like to see the amount of electricity/gas used between 00:00 and 01:00, 01:00 and 02:00, 02:00 and 03:00 etc. I'd like the Energy charts in HA to look the same as the energy charts in Octopus Energy's Explore My Energy Use page.

At present the entire previous days usage is shown as consumption in a single hour.

I'm not able to find any clear documentation on when Home Assistant decides to poll the API. I notice that most days the data is for 3am, but some days it is a different time.

Are you saying that HA can only treat the results of the poll as current? E.g. if the poll happens at 3am and returns any data, it will always be treated as the data for 3am, even if it is the data for a different time?

That makes sense if it was polling whether a light is off or on at a particular moment. I guess it would also make sense if the Octopus Energy API supplied a meter reading that could then be stored, rather than the consumption data for given periods of time.

I asked a question in HA Discord #devs_energy about this, but no response yet.

[BottlecapDave]

Sorry for the late response.

Home Assistant energy dashboard has been built for "live" data and bascially records the state value of an entity at the time it changes. From what I understand, Octopus Energy polls your meter on a daily basis, however there's no guarentee when the data becomes available. Therefore you can't get live data from Octopus Energy, only historic. And HA only supports live data in the energy dashboard, not historic.

Because of this, I made the decision to have the accumlative previous day as I thought this was the most valuable with the data being historic from Octopus and HA not supporting historic breakdowns in the energy dashboard.

[dunxd]

That makes sense. You have to work with the limitations of the system. I'll keep looking to see if there is a workaround or someone has a different idea for how to get data into HA from a daily updating API.

[townsmcp]

@dunxd the Bright app gives half hour updates. And there is an integration that can pull that data into HA: https://github.com/HandyHat/ha-hildebrandglow-dcc

if you want real time, there is also the CAD device you can have running (I haven’t bought this yet so don’t know how reliable etc it is): https://shop.glowmarkt.com/products/glow-stick

[dunxd]

@townsmcp That looks interesting. The Glow API seems to present the data in a similar way to the Octopus API - rolled up data made available once a day. Can you clarify if the integration populates the Energy graph fully in the hourly view (i.e. bars for each hour in a day)?

I'm reluctant to try it out as the Bright service doesn't have any privacy policies or descriptions of how they use my data prominent on their website or visible prior to signing up.

[townsmcp]

@dunxd Bright pulls the data half hourly because your meter reports back to the wan every half hour:

The zigbee stick effectively works the same as the IHD for real time reporting which can be tied into HA

[BottlecapDave]

Closing this issue as it looks like you have an alternative, and it feels like it is currently out of scope for what the integration can provide until HA provides more facilities (or Octopus provides data more frequently)

[dunxd]

Looks like someone is working on historic data support in HA for exactly this purpose - https://github.com/home-assistant/core/pull/56607

I don't see Bright as an alternative. Firstly, they don't seem to have any kind of privacy policy published, at least one this prospective user can easily find and read before signing up. Hildebrand are selling access to this data which they don't appear to be gathering in a responsible way. Happy to be proved wrong on this. Secondly, they are completely sold out of any of the devices for local reading of SMETS2 meters, which I have. It has been like that for a few months.

[BottlecapDave]

Looks like someone is working on historic data support in HA for exactly this purpose - home-assistant/core#56607

That's interesting. Once I come across some docs, I'll look at trying to integrate it if it fits what we're needing.

[jstammers]

If it helps, here's a rough example of how I've managed to process the results into the format that's required for async_add_external_statistics. I'm not familiar enough with the DataUpdateCoordinator class used here to figure out what the structure the result of update_method needs to have.

import asyncio
import os
from datetime import datetime

from homeassistant.components.recorder.statistics import StatisticData, StatisticMetaData, async_add_external_statistics
from homeassistant.core import HomeAssistant

from custom_components.octopus_energy.api_client import OctopusEnergyApiClient

mprn = ...
serial = ...
client = OctopusEnergyApiClient(os.environ["API_KEY"])

async def async_get_half_hourly_consumption():
    period_from = datetime.strptime("2021-12-01T00:00:00Z", "%Y-%m-%dT%H:%M:%S%z")
    period_to = datetime.strptime("2021-12-03T00:00:00Z", "%Y-%m-%dT%H:%M:%S%z")
    data = await client.async_get_electricity_consumption(mprn, serial, period_from, period_to)
    sum = 0
    last_reset = period_from
    statistics = []
    for item in data:
        consumption = item["consumption"]
        sum += consumption
        statistic = StatisticData(state=consumption,
                                  start=item["interval_start"],
                                  sum=sum,
                                  last_reset=last_reset)
        last_reset = item["interval_end"]
        statistics.append(statistic)
    return statistics

hass = HomeAssistant()
meta = StatisticMetaData(has_mean=False,
                         has_sum=True,
                         name="test_statistic", 
                         source="test_source",
                         statistic_id="test_statistic_id",
                         unit_of_measurement="kWh")

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
task = loop.create_task(async_get_half_hourly_consumption())
loop.run_until_complete(task)
statistics = task.result()
async_add_external_statistics(hass, meta, statistics)

[BottlecapDave]

That's very interesting. Was this based on any documentation that HA provide, or just trial and error?

[jstammers]

Hi, apologies for the late reply. This was just trial and error - there wasn't much in the way of documentation. It's possible that some has been added since I last checked