ML Based Load Prediction

[philjohn]

Is your feature request related to a problem? Please describe.

The load being predicted based on previous days is generally ok, but to take into account seasonality (and to add other factors) a dedicated ML based predictor might give better results.

Describe the solution you'd like

Meta published Prophet, which is a time series forecasting tool. There is a new project, based on Prophet, called Neural Prophet which builds an ML model based on previous usage data. You can also feed in other sensors so that the ML model can infer if they have a meaningful impact on the load forecast on a given day (e.g. outside temperature, days of sunlight, if it's a holiday etc.)

[springfall2008]

I love the idea, in fact I was talking about it myself with a friend last week. I fear it might however be quite compute intensive. I'll take a look at Prophet.

[springfall2008]

Well its easy to use, I managed this based on my Octopus data.

But, this of course isn't the right data to use as I need load not grid use, so I will need to export load from HA or the GE Portal. And car charging needs to be removed from the data somehow first.

import pandas as pd
from datetime import datetime, timedelta
from neuralprophet import NeuralProphet, set_log_level

TIME_FORMAT_OCTOPUS = "%Y-%m-%dT%H:%M:%S%z"

# Load the dataset from the CSV file using pandas
df = pd.read_csv("consumption.csv", skipinitialspace=True)
print(df.head())
df.drop('End', axis=1, inplace=True)
df = df.rename(columns={"Start": "ds", "Consumption (kWh)": "y"})

for index, row in df.iterrows():
    start = row['ds']
    start_time = datetime.strptime(start, TIME_FORMAT_OCTOPUS)
    row['ds'] = start_time

print(df.head())

# Disable logging messages unless there is an error
set_log_level("ERROR")

# Create a NeuralProphet model with default parameters
m = NeuralProphet()

# Fit the model on the dataset (this might take a bit)
metrics = m.fit(df)

# Create a new dataframe reaching 365 into the future for our forecast, n_historic_predictions also shows historic data
df_future = m.make_future_dataframe(df, n_historic_predictions=True, periods=365)

print(df_future.head())

# Predict the future
forecast = m.predict(df_future)

print(forecast.head())

# Write forecast to CSV
forecast.to_csv("forecast.csv", index=False)

[philjohn]

Yep - super easy to implement. One thing to bear in mind that yes, fitting the model is expensive, but doesn't need to be done every 5 minutes. Once a day would suffice, you can then just keep predicting using a saved model.

And yes, you're correct that you need a HA sensor that is just your house consumption - any other loads to be modelled.

Getting more complex with NP you can also feed in other sensors so that it can train any correlation between, e.g., weather forecast and load usage (seasonality built into NP is also a good enough proxy for this however).

[springfall2008]

When I get time I'll have a go and training it based on the last 7 days load data inside Predbat, like you say it doesn't need to happen every 5 minutes.

I think for a better data set Predbat may need to save the daily load data to a file as HA only keeps a short history.

[springfall2008]

So I ran into an issue, AppDeamon add-on runs on Alpine Linux which doesn't support torch and hence you can't use neuralprophet as-is.

I'm not sure on the implication of changing AppDeamon to a different linux distribution, I'll need to play around

[philjohn]

Ah, yes, it's due to MUSL libc missing a couple of key functions.

AWS got around it in their CLI docker container based on Alpine by essentially defining their own, but that would require patches to torch, which is a non starter.

I'll have a play as well, since AppDaemon can be installed with a simple pip install, it should be possible to use a different base for the container, but as you say, need to check the side effects.

[springfall2008]

This is now implemented as: https://github.com/springfall2008/predai