Allow configuring which datasets to regularly fetch

[pdcastro]

Describe the feature

During the integration config flow, allow the user to choose, I imagine through a list of check boxes, which datasets the API should regularly fetch. Datasets such as:

• [x] Current Consumption Data
• [x] Previous Consumption Data
• [x] Electricity Rates and Standing Charges
• [x] Gas Rates and Standing Charges
• [x] Greenness Forecast
• [x] Intelligent Dispatches and Settings
• [x] Saving Sessions
• [x] Wheel of Fortune

Currently I think that the config flow has only one configuration page / step:

The config flow could have two pages / steps, where the first page / step is the list of checkboxes where the user selects the datasets of interest, and the second page / step asks the additional questions (that are currently in the first page). Ideally, if possible, the second page / step would only ask questions that are relevant to the datasets chosen in the first page / step.

Expected behaviour

By default, all dataset checkboxes would be preselected, causing the integration to regularly fetch all data. However if the user selected, for example, only the “Intelligent Dispatches and Settings” checkbox, then only this dataset would be regularly fetched by the integration.

The list of datasets roughly matches the list of “coordinators” in the custom_components/octopus_energy/coordinators/ folder of the integration source code.

Use Case

Some users are only interested in a subset of the available datasets. For example, a reason why many users (myself included) prefer the megakid/ha_octopus_intelligent integration over this integration is that it only fetches data relevant to the Intelligent Go tariff, which is the only data these users have a use for.

Pointlessly querying, downloading and storing data that won’t be used has several consequences:

• Users may occasionally hit the Octopus API request limit (100 req/h) and be prevented from making queries that they care about, at least in part because of this integration’s queries that they don’t care about. It has happened to me.
• As the FAQ entry How often is data refreshed? points out, the refresh rates were chosen ”to try and not overload the API [...] - Octopus Energy estimate that ~95% of their traffic comes mainly from this integration.” Collectively, such cloud API processing load has an environmental cost too: Datacentres are energy and water hungry as we increasingly read in the news, for example globally or in southern England.
• The data is stored locally in the Home Assistant server, sometimes backed by an SD Card that physically wears and eventually fails — e.g. commonly on a Raspberry Pi.
• The data download consumes 4G or broadband bandwidth.

Confirmation

• [X] By submitting this feature request, you agree that you have read the documentation and confirmed it does not already exist
• [X] I am willing/able to help contribute to the solution of this feature

[BottlecapDave]

Hello and sorry for the late response.

By default, all dataset checkboxes would be preselected, causing the integration to regularly fetch all data. However if the user selected, for example, only the “Intelligent Dispatches and Settings” checkbox, then only this dataset would be regularly fetched by the integration.

The list of datasets roughly matches the list of “coordinators” in the custom_components/octopus_energy/coordinators/ folder of the integration source code.

Unfortunately the data requested by sensors is not so cut a dry. For example peak/off peak consumption sensors require rate information in addition to consumption data in order to determine which bucket the consumption data belongs to. Cost sensors require consumption data, rate data and standing charge data. Trying to structure the config to support this in addition to presenting this to a user in a way that is easy to understand would take up most of my time.

Some users are only interested in a subset of the available datasets. For example, a reason why many users (myself included) prefer the megakid/ha_octopus_intelligent integration over this integration is that it only fetches data relevant to the Intelligent Go tariff, which is the only data these users have a use for.

By using both this and another intelligent based integration, you will not be helping yourself with rate limits. I'm always interested to hear what that integration provides over this one, but not sure of this argument. Are you saying this integration provides too much?

The data is stored locally in the Home Assistant server, sometimes backed by an SD Card that physically wears and eventually fails — e.g. commonly on a Raspberry Pi.

To my knowledge HA doesn't persist information to disk straight away and only saves sensors that have changed, otherwise sensors that are far more chatty (e.g. zigbee sensors) would wreck the SD card long before this integration does.

Users may occasionally hit the Octopus API request limit (100 req/h) and be prevented from making queries that they care about, at least in part because of this integration’s queries that they don’t care about. It has happened to me.

OE have different rate limits for different API calls and to my knowledge are independent of each other (for example current consumption data has a lower rate limit than intelligent data). For the areas that are prone to sensitive limits, configuration is provided as you mentioned. For other areas, they are well below the known rate limits.

The integration only requests data that it needs for the provided sensors to function. Due to the HA structure, if you disable all entities for a given area (e.g. greener days) then it won't request the data because nothing has subscribed to it. For people not familar with HA this is probably unknown, so I'll update the FAQ with this information.

Therefore I have no plans to include this functionality due to the complexity and the interweaving of the data required for sensors to work (which will inevitably lead to bugs) and that HA already provides this indirectly by disabling entities.