Future Agile rates suggestion

[slopemad]

At the moment, Predbat assumes unpublished Agile rates are the same as today's rates, and there is an offset function to calculate tomorrow. Trying to think of how it might be better to do this based on Octopus Watch or the agilepredict twitter account suggestions.

I can see that the overnight rates tonight could be about 18p (whereas last night they were down to 0.5p), so I have created a rates_import_override to deal with this: rates_import_override:

• date: '2023-10-31' start: '00:00:00' end: '07:00:00' rate: 18

But I'll need to remember to delete this at 4pm when tomorrow's Agile rates are out. The rates offset would be a very crude tool, because the rates beyond overnight will be broadly similar to today anyway, hence creating an override. If I used the rate offset to add 18p, it would likely hike the Low Threshold very high.

But... I only want this override to apply as long as the real Octopus rates are unpublished. Perhaps there could be an alternative to rates_import_override which only applies until the actual Agile rates are published, and then the actual rates override the override? The 'date' field is also probably important, as I wouldn't necessarily want my assumed rates for tomorrow to apply for the day after also?

This will help in situations where you have rates tomorrow which differ wildly from today.

[springfall2008]

This seems like a good idea to me, I could do a soft override that only applies to unpublished rates. I think I could also support a offset option in this case e.g. +18 rather than 18.

[gcoan]

I’m very new to Agile so don’t have a lot of experience of how the rates change over time. I’d like predbat to generally get the right direction of the rates rather than need to obsess over a precise prediction. It would seem to me that:

• evening peak is typically going to be the same all week, except perhaps for really cold periods, so that could default to previous day
• there will be lower prices overnight, rising during the day, price being based on day of week (weekends lower) and weather/solar forecast- so copying yesterdays rate with a +/- offset that could be set programmatically based on day of week/solar/wind forecast if you wanted to get that sophisticated

Geoffrey

On 30 Oct 2023, at 10:49, Trefor Southwell @.***> wrote:

 This seems like a good idea to me, I could do a soft override that only applies to unpublished rates. I think I could also support a offset option in this case e.g. +18 rather than 18.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

[kernow]

I contacted the guy who runs Octopus Watch so see if he had an API for the 48 hour prediction that could be hooked into. Unfortunately it's not something that is available to the public, or paid. Would have been an ideal solution

[slopemad]

I guess how far into the future a user would like Predbat to have some knowledge of rates depends on the battery:daily usage ratio. As we go into winter, my ratio is pretty small, so the view to what rates might look like overnight is important, but I'm not so fussed about what the rates look like at tomorrow breakfast time, that can happily wait until the real rates come out at 4pm. Someone with a higher battery:demand ratio might want a longer term view. A soft override set to a particular time (either as an offset, or a fixed figure) would be ideal.

How I see it:

rate_soft_override:

• start: '01:00:00' end: '05:00:00' offset: -1 (or rate: 14 to fix the rate rather than offset it)

Would be more useful to me than the future rate offset on it's own. If I could map that to a HA sensor

offset: number.some_ha_number

Even better - saves editing apps.yaml every day :-). Other users may prefer to create additional rate offset period, if they have more battery capacity.

[gcoan]

For me personally it’d be nice to have a better “guess” of the next day rates but it’s not super critical.

My daily house load (with an ASHP) is 2-3 times my 14.8kW battery capacity so I’m typically charging to full overnight and depending on solar and rates doing further charging in the morning or afternoon. So a long look forward doesn’t really add a lot for me. In fact I’ve reduced the forecast_hours down to 36 hours from 48 as having 2 days of future dated agile rates in the predbat charging plan was just cluttering the dashboard

On 6 Nov 2023, at 21:11, slopemad @.***> wrote:

 I guess how far into the future a user would like Predbat to have some knowledge of rates depends on the battery:daily usage ratio. As we go into winter, my ratio is pretty small, so the view to what rates might look like overnight is important, but I'm not so fussed about what the rates look like at tomorrow breakfast time, that can happily wait until the real rates come out at 4pm. Someone with a higher battery:demand ratio might want a longer term view. A soft override set to a particular time (either as an offset, or a fixed figure) would be ideal.

How I see it:

rate_soft_override:

start: '01:00:00' end: '05:00:00' offset: -1 (or rate: 14 to fix the rate rather than offset it) Would be more useful to me than the future rate offset on it's own. If I could map that to a HA sensor

offset: number.some_ha_number

Even better - saves editing apps.yaml every day :-). Other users may prefer to create additional rate offset period, if they have more battery capacity.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[slopemad]

So, here's what I've been doing the past few days to model 'day ahead' pricing for Agile, and it's there or there abouts. How to apply this in Batpred is another matter.

So I take the N2EX Next Day pricing from Nordpool: https://www.nordpoolgroup.com/en/Market-data1/GB/Auction-prices/UK/Hourly/?view=table Tomorrow's hourly prices are out at 10am in the morning, making sure I get the GBP prices rather than EUR prices ;-) Convert the hourly MWh prices to kWh prices by dividing by 1000 Apply the Agile formula, i.e. (1.05 min((kWh price 2.2) + P, 95)), I use P = 12 (only applies 4pm-7pm) and then add override rates (see below). This goes in around 10am, and gets removed around 4pm when the actual Agile rates are out.

How the N2EX rates can be scraped into Predbat and rates applied on an automagic basis, I have no idea. With tonight's overnight rates being quite pricy, this led to more charging this afternoon to counter it.

  rates_import_override:
    - date: '2023-11-09'
      start: '23:00:00'
      end: '00:00:00'
      rate: 20
    - date: '2023-11-10'
      start: '00:00:00'
      end: '01:00:00'
      rate: 20
    - date: '2023-11-10'
      start: '01:00:00'
      end: '02:00:00'
      rate: 22
    - date: '2023-11-10'
      start: '02:00:00'
      end: '06:00:00'
      rate: 20
    - date: '2023-11-10'
      start: '06:00:00'
      end: '07:00:00'
      rate: 23
    - date: '2023-11-10'
      start: '07:00:00'
      end: '08:00:00'
      rate: 26
    - date: '2023-11-10'
      start: '08:00:00'
      end: '09:00:00'
      rate: 29
    - date: '2023-11-10'
      start: '09:00:00'
      end: '10:00:00'
      rate: 30
    - date: '2023-11-10'
      start: '10:00:00'
      end: '11:00:00'
      rate: 25
    - date: '2023-11-10'
      start: '11:00:00'
      end: '12:00:00'
      rate: 20
    - date: '2023-11-10'
      start: '12:00:00'
      end: '13:00:00'
      rate: 19
    - date: '2023-11-10'
      start: '13:00:00'
      end: '15:00:00'
      rate: 20
    - date: '2023-11-10'
      start: '15:00:00'
      end: '16:00:00'
      rate: 21
    - date: '2023-11-10'
      start: '16:00:00'
      end: '17:00:00'
      rate: 38
    - date: '2023-11-10'
      start: '17:00:00'
      end: '18:00:00'
      rate: 40
    - date: '2023-11-10'
      start: '18:00:00'
      end: '19:00:00'
      rate: 37
    - date: '2023-11-10'
      start: '19:00:00'
      end: '20:00:00'
      rate: 23
    - date: '2023-11-10'
      start: '20:00:00'
      end: '21:00:00'
      rate: 22
    - date: '2023-11-10'
      start: '21:00:00'
      end: '22:00:00'
      rate: 21
    - date: '2023-11-10'
      start: '22:00:00'
      end: '23:00:00'
      rate: 20

[springfall2008]

Seems like a good idea, I'll see if it's possible to work something up when I get time

[fboundy]

I tried to scrape these a while ago for just this purpose but they see to have the relevant table encrypted or hidden somehow. Beautiful Soup got nowhere with it. On 9 Nov 2023 at 16:57 +0000, Trefor Southwell @.***>, wrote:

Seems like a good idea, I'll see if it's possible to work something up when I get time — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[slopemad]

The data appears to be in here... https://www.nordpoolgroup.com/api/marketdata/page/325?currency=GBP

[fboundy]

Might have another go then. On 11 Nov 2023 at 09:15 +0000, slopemad @.***>, wrote:

The data appears to be in here... https://www.nordpoolgroup.com/api/marketdata/page/325?currency=GBP — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[kernow]

It would be prudent to have a single source take this data periodically, maybe translate it into the relevant values, and then publish it somewhere else that each predbat instance can download from. If 100's of computers suddenly start to pull down this data every day they will soon shut it off if it's not something intended for public consumption in this way.

This would mean having a service separate from predbat, could maybe use something like fly.io or heroku on their free plans?

[RobinCu]

It depends on how the data is translated but I'd be cautious that using the data and publishing it in a public location (even if the Agile formula was applied) it could fall foul of the data providers' redistribution license terms. Also, I've just checked and the Nord Pool Group website gets 7 million users a month, so I think we'd go unnoticed until Trefor finalises his plans for world domination!

[fboundy]

If you look at Nordpool’s Ts and Cs they forbid automated downloading IIRC On 13 Nov 2023 at 15:19 +0000, RobinC @.***>, wrote:

It depends on how the data is translated but I'd be cautious that using the data and publishing it in a public location (even if the Agile formula was applied) it could fall foul of the data providers' redistribution license terms. Also, I've just checked and the Nord Pool Group website gets 7 million users a month, so I think we'd go unnoticed until Trefor finalises his plans for world domination! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[slopemad]

Yeah I have seen that. I just have a look to see if there's already a Nordpool integration for Home Assistant and it turns out there is... (But it doesn't necessarily pick up the GB prices in it's current form).

https://github.com/custom-components/nordpool

[gcoan]

No information as to why GB not available in the nord pool HA integration so I've raised a discussion item to find out whether it can be added.

https://github.com/custom-components/nordpool/discussions/355 On 13 Nov 2023 at 15:53 +0000, slopemad @.***>, wrote:

Yeah I have seen that. I just have a look to see if there's already a Nordpool integration for Home Assistant and it turns out there is... (But it doesn't necessarily pick up the GB prices in it's current form). https://github.com/custom-components/nordpool — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[fboundy]

It’s something to do with Brexit. GB (NI is different) is no longer part of the EU balancing mechanism. Something along those lines anyway. On 13 Nov 2023 at 16:18 +0000, Geoffrey Coan @.***>, wrote:

No information as to why GB not available in the nord pool HA integration so I've raised a discussion item to find out whether it can be added.

https://github.com/custom-components/nordpool/discussions/355 On 13 Nov 2023 at 15:53 +0000, slopemad @.***>, wrote:

Yeah I have seen that. I just have a look to see if there's already a Nordpool integration for Home Assistant and it turns out there is... (But it doesn't necessarily pick up the GB prices in it's current form). https://github.com/custom-components/nordpool — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

[springfall2008]

I've been playing with this and I'm able to download the data and cache it in Predbat.

I was wondering how to use it, would just a % change day to day feed directly into Agile rates? Do you apply some cap and base price adjustment?

It's here to play with but only tested as far as downloading the data:

https://github.com/springfall2008/batpred/tree/future-energy-rates

[slopemad]

So if we have the hourly data, then I'd have thought we may as well apply the magic Agile formula to each hour and use that in predbat.rates until the official prices are available? Whilst the actual half hourly prices from Octopus will vary, they're not a million miles. The magic formula being the Wholesale MWH rate (W) 0.001 (convert to kWh) VAT (1.05) 2.2 (the magic Agile multiplier), and add P (12p, between 4pm and 7pm) So - (W 0.001 1.05 2.2) + P which should be no more than £1 :-)

For example, the originally published wholesale rate for tonight, 23.00-0000 is £62.50, which is translated by the formula to 14p (and the real published prices are 15.44p and 10.5p for each half hour slot) - so may as well just use a rate of 14p for that hour until the real Octopus prices are out? This should be close enough for those with high battery to usage ratios to get a good plan together for the afternoon, before the real rates are out.

[springfall2008]

What is P outside 4-7pm Period?

[slopemad]

P is zero, except between 4pm and 7pm when it is between 11 and 14p (But I have no way of knowing which figure it is, so I use 14). It's basically the peak-time period. https://octopus.energy/blog/agile-pricing-explained/

[springfall2008]

This was all going well up until the point where I tried to test it this morning and found that tomorrows rates aren't published yet either. Is this data really ahead of Octopus?

[RobinCu]

I think tomorrow's rates are published just before 11am CET, so should be updated shortly.

[slopemad]

They're published at around 10am UK time, gives a 6 hour sneak preview into tomorrow's Octopus rates. (Make sure currency is GBP rather than EUR).

[slopemad]

The wholesale rates are available now and these are the sort of estimated Agile prices you want to see from it.

[RobinCu]

That's close to Octopus Watch's forecasts (which are usually +/- 1p):

It would certainly be a very useful addition to know whether to charge pre-4 pm or wait until overnight.

[springfall2008]

Great, I'm nearly there now just fixing the arithmetic

[springfall2008]

Please have a try on the code on main and see if it works for you, the example configuration is commented out in the template apps.yaml but can be set for import and export, e.g. for Agile use this:

  # Nordpool market energy rates
  futurerate_url: 'https://www.nordpoolgroup.com/api/marketdata/page/325?currency=GBP'
  futurerate_adjust_import: True
  futurerate_adjust_export: True
  futurerate_peak_start: "16:00:00"
  futurerate_peak_end: "19:00:00"
  futurerate_peak_premium_import: 14
  futurerate_peak_premium_export: 6.5

[fboundy]

It's a nice feature but personally I won't be using it given the explicit prohibition on their website. If this were my repo I would be very wary of putting something out there which facilitated others doing that but obviously it's your call.

[gcoan]

Plan with 7.12.1:

Logfile entries:

Fetching futurerate data from https://www.nordpoolgroup.com/api/marketdata/page/325?currency=GBP
2023-11-14 11:02:40.026989 INFO pred_bat: Future rates - peak rate is 960 - 1140 minutes premium import 14 export 6
2023-11-14 11:02:40.074371 INFO pred_bat: Loaded 192 datapoints of futurerate analysis
2023-11-14 11:02:40.098295 INFO pred_bat: Predicted future rates: ['0 => 14.79 / 6.08', '60 => 15.34 / 6.31', '120 => 12.12 / 4.98', '180 => 11.55 / 4.75', '240 => 9.89 / 4.07', '300 => 13.27 / 5.46', '360 => 19.57 / 8.05', '420 => 20.11 / 8.27', '480 => 20.71 / 8.52', '540 => 19.42 / 7.98', '600 => 20.8 / 8.56', '660 => 22.08 / 9.08', '720 => 21.23 / 8.73', '780 => 20.79 / 8.55', '840 => 21.81 / 8.97', '900 => 20.13 / 8.28', '960 => 41.96 / 17.21', '1020 => 43.84 / 17.98', '1080 => 38.44 / 15.76', '1140 => 21.0 / 8.64', '1200 => 21.22 / 8.72', '1260 => 20.79 / 8.55', '1380 => 19.63 / 8.07']

plan with new 'main' version:

Seems to work. I see it caches the previous call so doesn't make repeat downloads

[slopemad]

So this is in the log:

2023-11-14 11:01:25.422495 INFO pred_bat: Fetching futurerate data from https://www.nordpoolgroup.com/api/marketdata/page/325?currency=GBP
2023-11-14 11:01:25.903003 INFO pred_bat: Future rates - peak rate is 960 - 1140 minutes premium import 14 export 0
2023-11-14 11:01:25.948345 INFO pred_bat: Loaded 192 datapoints of futurerate analysis
2023-11-14 11:01:25.982604 INFO pred_bat: Predicted future rates: ['0 => 14.79 / 6.08', '60 => 15.34 / 6.31', '120 => 12.12 / 4.98', '180 => 11.55 / 4.75', '240 => 9.89 / 4.07', '300 => 13.27 / 5.46', '360 => 19.57 / 8.05', '420 => 20.11 / 8.27', '480 => 20.71 / 8.52', '540 => 19.42 / 7.98', '600 => 20.8 / 8.56', '660 => 22.08 / 9.08', '720 => 21.23 / 8.73', '780 => 20.79 / 8.55', '840 => 21.81 / 8.97', '900 => 20.13 / 8.28', '960 => 41.96 / 11.21', '1020 => 43.84 / 11.98', '1080 => 38.44 / 9.76', '1140 => 21.0 / 8.64', '1200 => 21.22 / 8.72', '1260 => 20.79 / 8.55', '1380 => 19.63 / 8.07']

This data as collected is based on today's wholesale prices rather than tomorrow's?

[RobinCu]

So this is in the log:

2023-11-14 11:01:25.422495 INFO pred_bat: Fetching futurerate data from https://www.nordpoolgroup.com/api/marketdata/page/325?currency=GBP
2023-11-14 11:01:25.903003 INFO pred_bat: Future rates - peak rate is 960 - 1140 minutes premium import 14 export 0
2023-11-14 11:01:25.948345 INFO pred_bat: Loaded 192 datapoints of futurerate analysis
2023-11-14 11:01:25.982604 INFO pred_bat: Predicted future rates: ['0 => 14.79 / 6.08', '60 => 15.34 / 6.31', '120 => 12.12 / 4.98', '180 => 11.55 / 4.75', '240 => 9.89 / 4.07', '300 => 13.27 / 5.46', '360 => 19.57 / 8.05', '420 => 20.11 / 8.27', '480 => 20.71 / 8.52', '540 => 19.42 / 7.98', '600 => 20.8 / 8.56', '660 => 22.08 / 9.08', '720 => 21.23 / 8.73', '780 => 20.79 / 8.55', '840 => 21.81 / 8.97', '900 => 20.13 / 8.28', '960 => 41.96 / 11.21', '1020 => 43.84 / 11.98', '1080 => 38.44 / 9.76', '1140 => 21.0 / 8.64', '1200 => 21.22 / 8.72', '1260 => 20.79 / 8.55', '1380 => 19.63 / 8.07']

This data as collected is based on today's wholesale prices rather than tomorrow's?

I was thinking the same. I've not done an hour-by-hour comparison but it does seem to look more like today's.

[springfall2008]

It's all the data from midnight this morning until the data runs out tomorrow night. It computes the difference between today and tomorrow and adds that to the real Agile rate for today to get to tomorrow.

So 0 is 0 minutes from midnight today, for tomorrow you have to scroll right but I just noticed I didn't log all the rates so it stops at midnight tonight.

[RobinCu]

I've made a start at calculating P (peak premium import) by region, just as a proof of concept. Using today's 17:00 - 17:30 market rate and comparing it to 17:00 Agile by region). I thought it could be added to the read me.

What's odd is that one region already is exceeding the quoted range of 11 - 14 p. Maybe Octopus are rounding down or simplifying the formula in their explainer. Or my maths is wrong!

If you think it will be useful, I'll finish the other regions. Or does the computation of the difference between today and tomorrow compensate enough for regional variance in P? i.e. am I adding unnecessary complexity to the read me?

[springfall2008]

I think you are right and it's a good suggestion, but yes the adjustment is made based on price differences so in fact P will have no impact to tomorrows data assuming P doesn't change.

The Agile export multiplier of 0.95 I think can change a little it's not fixed like the 2.2 for import but again I think it's close enough as I'm just looking for adjustments from real rates

[slopemad]

So this is what my plan looks like for tomorrow. Will be interesting to compare with the actual Octopus prices when they come out at 4pm.

[gcoan]

R/e the nordpool terms and conditions. I suspect that nordpool is trying to encourage people to use their paid services, and as a commercial company, why wouldn’t they.

But given there is already a home assistant add-on, YouTube videos and blog articles published about how to pull this data into home assistant, and the website isn’t paywall protected in any way not even a recaptcha, it seems to me that nordpool are not all that bothered about casual users like us.

As a suggestion, if this goes into a future release that the readme or apps.yaml points the user to nordpool’s website for terms and conditions of using their data then we’re covered.

On 14 Nov 2023, at 11:09, fboundy @.***> wrote:

 It's a nice feature but personally I won't be using it given the explicit prohibition on their website. If this were my repo I would be very wary of putting something out there which facilitated others doing that but obviously it's your call.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[springfall2008]

Yes @gcoan makes sense, in fact I didn't mention Nordpool in the code itself and left it to the user to configure it so they can't blame us if the user decides to use that service.

[fboundy]

So would you leave it out of the released config.yaml? On 14 Nov 2023 at 11:58 +0000, Trefor Southwell @.***>, wrote:

Yes @gcoan makes sense, in fact I didn't mention Nordpool in the code itself and left it to the user to configure it so they can't blame us if the user decides to use that service. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[springfall2008]

So would you leave it out of the released config.yaml? … On 14 Nov 2023 at 11:58 +0000, Trefor Southwell @.>, wrote: Yes @gcoan makes sense, in fact I didn't mention Nordpool in the code itself and left it to the user to configure it so they can't blame us if the user decides to use that service. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

No I was going to leave it commented out so Predbat doesn't ship with it enabled, it's only useful for Agile anyhow.

[fboundy]

If it's commented out it seems to cause an error:

12:54:01 WARNING pred_bat: ------------------------------------------------------------
12:55:01 WARNING pred_bat: ------------------------------------------------------------
12:55:01 WARNING pred_bat: Unexpected error in worker for App pred_bat:
12:55:01 WARNING pred_bat: Worker Ags: {'id': '5a0325f67291483bb5d3197234145ad1', 'name': 'pred_bat', 'objectid': '97f2dd0523ce4bb09f87af3f175bb96c', 'type': 'scheduler', 'function': <bound method PredBat.run_time_loop of <predbat.PredBat object at 0x7f845e48d0>>, 'pin_app': True, 'pin_thread': 0, 'kwargs': {'interval': 300, 'random_start': 0, 'random_end': 0, '__thread_id': 'thread-0'}}
12:55:01 WARNING pred_bat: ------------------------------------------------------------
12:55:01 WARNING pred_bat: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/threading.py", line 1022, in worker
    funcref(self.AD.sched.sanitize_timer_kwargs(app, args["kwargs"]))
  File "/config/appdaemon/apps/batpred/predbat.py", line 8902, in run_time_loop
    raise e
  File "/config/appdaemon/apps/batpred/predbat.py", line 8898, in run_time_loop
    self.update_pred(scheduled=True)
  File "/usr/lib/python3.11/site-packages/appdaemon/adbase.py", line 35, in f_app_lock
    return f(*args, **kw)
           ^^^^^^^^^^^^^^
  File "/config/appdaemon/apps/batpred/predbat.py", line 8468, in update_pred
    self.fetch_sensor_data()
  File "/config/appdaemon/apps/batpred/predbat.py", line 7919, in fetch_sensor_data
    self.future_energy_rates_import, self.future_energy_rates_export = self.futurerate_analysis()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 0)

[springfall2008]

Good spot, I pushed a fix for that now

[RobinCu]

Looking at the just released prices, I'd say it's done a good job. Obviously, there are few slots where the bidding has flipped the pricing between two halves within an hour but that would have almost no impact on the overall charging cost.

[RobinCu]

Looking at the just released prices, I'd say it's done a good job. Obviously, there are few slots where the bidding has flipped the pricing between two halves within an hour but that would have almost no impact on the overall charging cost.

Here's my pre-4 pm forecast power graph:

And here with actual rates:

I think I can retire my metric future offset input!

[slopemad]

Looking good from here, there are a few slots where it might be a couple of pence out but there's nothing that can be done about that, it's the nature of the forecasting beast. The general trend is there, and whilst tomorrow's prices are not much different to today anyway, where we get days where the prices are going to wildly differ from one day to the next, this function really will come into its own. It's a lot less blunt than the future offset metric.

[gcoan]

Looks like predbat caches the nordpool data for an hour?

At 15:00:

2023-11-14 15:00:01.851873 INFO pred_bat: Return cached futurerate data for https://www.nordpoolgroup.com/api/marketdata/page/325?currency=GBP age 59.983333333333334 minutes
2023-11-14 15:00:01.852966 INFO pred_bat: Future rates - peak rate is 960 - 1140 minutes premium import 14 export 6
2023-11-14 15:00:01.872615 INFO pred_bat: Loaded 192 datapoints of futurerate analysis
2023-11-14 15:00:01.886717 INFO pred_bat: Predicted future rates: ['0 => 14.79 / 6.08', '60 => 15.34 / 6.31', '120 => 12.12 / 4.98', '180 => 11.55 / 4.75', '240 => 9.89 / 4.07', '300 => 13.27 / 5.46', '360 => 19.57 / 8.05', '420 => 20.11 / 8.27', '480 => 20.71 / 8.52', '540 => 19.42 / 7.98', '600 => 20.8 / 8.56', '660 => 22.08 / 9.08', '720 => 21.23 / 8.73', '780 => 20.79 / 8.55', '840 => 21.81 / 8.97', '900 => 20.13 / 8.28', '960 => 41.96 / 17.21', '1020 => 43.84 / 17.98', '1080 => 38.44 / 15.76', '1140 => 21.0 / 8.64', '1200 => 21.22 / 8.72', '1260 => 20.79 / 8.55', '1380 => 19.63 / 8.07']

Then at 15:05, looks like it downloads again:

2023-11-14 15:05:02.072650 INFO pred_bat: Fetching futurerate data from https://www.nordpoolgroup.com/api/marketdata/page/325?currency=GBP
2023-11-14 15:05:03.281304 INFO pred_bat: Future rates - peak rate is 960 - 1140 minutes premium import 14 export 6
2023-11-14 15:05:03.303299 INFO pred_bat: Loaded 192 datapoints of futurerate analysis
2023-11-14 15:05:03.328027 INFO pred_bat: Predicted future rates: ['0 => 14.79 / 6.08', '60 => 15.34 / 6.31', '120 => 12.12 / 4.98', '180 => 11.55 / 4.75', '240 => 9.89 / 4.07', '300 => 13.27 / 5.46', '360 => 19.57 / 8.05', '420 => 20.11 / 8.27', '480 => 20.71 / 8.52', '540 => 19.42 / 7.98', '600 => 20.8 / 8.56', '660 => 22.08 / 9.08', '720 => 21.23 / 8.73', '780 => 20.79 / 8.55', '840 => 21.81 / 8.97', '900 => 20.13 / 8.28', '960 => 41.96 / 17.21', '1020 => 43.84 / 17.98', '1080 => 38.44 / 15.76', '1140 => 21.0 / 8.64', '1200 => 21.22 / 8.72', '1260 => 20.79 / 8.55', '1380 => 19.63 / 8.07']

I assume that once nordpool creates the data at around 11am CET, its static for the rest of the day. Can we extend the caching to reduce repeated downloads from the nordpool site?

and a |int missing from the logging of the cache age as well

[springfall2008]

Yes, I think I could change the caching somewhat based on this.

[slopemad]

The prices on the Nordpool site, once released, are static for the day. They're only published once a day at 9.59am UK time. There is a chance they may be published late though. So maybe it wants to try and pull it at 10.00, and if it doesn't pull fresh data, then have another go at, say, 10.10, 10.20, 10.30... until it does get fresh data? And I assume there is graceful handling of the scenario where the Nordpool site doesn't return data, or doesn't get a "200 OK" response from the server?

[slopemad]

The main thing I observed this morning is that the rates cache downloaded at 09.45 and therefore didn't redownload again until 10.45 (whereas the new rates came out at 10). Not a problem for me as the plan was a full charge by 3.30pm anyway because I don't have enough battery capacity anyway, but someone with a better battery:load ratio (or me in summer) might see different results, especially with an anticipated 6p difference between today's and tomorrow's rates, which makes a huge difference to the low rate threshold.

However, the rates it has forecast for tomorrow look similar to those I calculated myself, which is excellent (although it doesn't appear to be calculating the 23:00 - 00:00 windows tonight?)

[slopemad]

OK, so when tomorrow's Nordpool rates went into HA, all the charging plans for today disappeared out of the window, because they caused they caused the Low Threshold to drop substantially, meaning that the plan suddenly involved a flat battery before today's evening peak. (I was going to do a manual charge, but it turns out there was substantially more solar than forecast and so I'll probably be OK).

This chart shows the drop of the threshold about 10ish, dropping below the afternoon's low rates. It's a 'feature'.

And even increasing the number of windows from 40 to 48 and even 56 didn't bring the low threshold up enough to capture the afternoon's low rate.

Perhaps there could be some kind of alert, but I'm not sure what the conditions might be for it. Or perhaps if it detects the battery getting to zero (perhaps within the next n hours, where n could be 6, 8, 10, 12, I don't know?), it could look back to see if the cheapest slot between now and the zero SOC would be cheaper than letting it use the rate at which the zero SOC is reached, even if that cheapest slot is above the low threshold? (In this case, the threshold dropped from 22.3 to 19.4p, meaning this lunchtime's 21.6p rate suddenly became above the threshold, so the plan changed to not charge at this afternoon low, but I'd have rather Predbat decided "Hey, I'll use that 21.6p rate, because that's better than a flat battery at 40+ pence".

Alternatively, perhaps the Low Threshold calculation should be based on "the next 24 hours" rather than the full 48 hours plan? (again, the size of this window could be user adjustable based on their battery:daily load ratio?)

[springfall2008]

In a 48 hour plan there can only be 96 windows at the most.

I'm actually wondering if I can get rid of the windows setting soon and just hard-wire it back to 96. It was done for performance reasons but I'm not sure if it's still needed.

That said I can see the value in having higher rate windows in the near term, rather than just the N lowest ones, I'll think about that too.