[solcast] "all-day" API request optimization

Matze2 commented 1 year ago

As far as I understood the solcast code, it downloads the latest forecast data only between sunrise and sunset (except of force mode). It might be useful to have also updates during the night to be able to plan power usages for the next day, e.g. to fill a washing machine and to program a timer for the next day. This is especially needed in winter time where the sunrise is after a working day usually starts. Currently there is only quite old forecast data from the last day before sunset available.

My suggestion to improve the usage of solcast is as follows:

The solcast API usage is distributed over the whole day - not only between sunrise and sunset
The intervals for the night phase are longer by a factor

Some ideas to choose the enlargement factor for intervals between sunset and sunrise: I just calculated the intervals with extreme examples (8-hour day and 16-hour day) and different factors. A factor of 4, i.e. download interval at night is 4 times larger than during the day, makes most sense to me.

Here are some examples for 48 API requests distributed over the day

Factor=2, 16-hours-day: Download Intervals in Minutes: 25 day / 50 night
Factor=2, 8-hours-day: Download Intervals in Minutes: 20 day / 40 night
Factor=3, 16-hours-day: Download Intervals in Minutes: 16 day / 48 night
Factor=3, 8-hours-day: Download Intervals in Minutes: 23 day / 69 night
Factor=4, 16-hours-day: Download Intervals in Minutes: 15 day / 60 night
Factor=4, 8-hours-day: Download Intervals in Minutes: 22 day / 88 night

The formula to calculate the above intervals

day_period = sunset - sunrise
night_period = full_day - day_period
day_interval = (night_period / factor - day_period) / api_requests
night_interval = day_interval * factor

I already implemented this in a fork which runs since today. After some testing I can create a pull request if there is interest.

Matze2 commented 1 year ago

Just to get an idea how it looks like: each tick means an API download from solcast. The first chart is from today up to now, the second one is a few days old with the existing code. grafik

StefaE commented 1 year ago

Hi Matthias, indeed that's a constraint of the current implementation and an extension is probably a good thing. I'd like to stay compatible with crontab calls to PVForecasts - which means, I get control every so often, eg. every 15min. Implementing a 23min interval then looks a bit awkward (but possible) ... But I'm open for a pull request.

Other options include:

(especially in winter) extend the day_period such that all credits are spent (eg. today, on a short winter day, I've only used 32 credits ...
add / subtract a fixed value to sunrise, sunset (eg. 3h)

Matze2 commented 1 year ago

Hi Stefan,

for my implementation I use a modified crontab with 5-minute periods. Otherwise it would not make sense to calculate exact values. Now that I dig more in the code I see that you round timestamps more often than I thought, e.g. also for issue time which leads to an anomaly in the first daytime call in my implementation. If you really want to stick to the 15-minutes period then my implementation would even make it worse since the periods are not 15 minutes anymore. Can you explain why this kind of fixed 15-minute setup including crontab is so important for you?

Just some notes to your comments. Extending the day_period will not work in summer since there are no credits left during the day. It is not even enough for 15-minute periods Adding always 3h to sunset and sunrise will also not work in summer, late spring and early autumn. You would run out of credits.

Anyway, I will publish my current implementation as pull request. Just for better discussion. It still contains some debug print() calls which I needed to understand the current code better. If it is against your philosophy I have also no problem to keep it in my fork.

And here is the download distribution of yesterday: grafik

StefaE commented 1 year ago

Hi Matthias, had not yet a chance to look at the code. Will do so over the weekend (and also need review my code, I guess...) The "importance" of a 15min is rooted in the update frequency of solcast: check their overview chart - implicitly, I obviously focus on the European zone.

StefaE commented 1 year ago

Hi Matthias, thanks for fixing the bug on the parenthesis - my bad :-(

Whilst your pull request looks clean in principle, I'm still struggling with the alignment

Solcast strictly has periodEnd = xx:00 and xx:30 (assuming Europe/Africa). So, we should call shortly after updates are done: then the first returned period is just the immediate next 30min and most accurate.
if we call at xx:15 and xx:45, the first period returned is actually 15min estimated actual data (xx:00 .. xx:15) and 15min forecast (xx:15 .. xx:30). [Later periods, eg. ((xx+1h):00, etc.) are also updated, but whether they are significantly more accurate, I doubt - I was meaning for a long while to publish some correlation data for my site... but didn't come around wrapping it up]

It might actually be sufficient to call Solcast every 30min (which works for the full 24h with 50 credits).

We could still go to 15min intervals on an extended day, something like this:

    sunrise        = mySun['sunrise'] + timedelta(hours=preSunRise)
    sunset         = mySun['sunset']  + timedelta(hours=postSunSet)

What I don't like in my current implementation is that I start on next 15min interval after sunrise ... it would make better use of the data, if that was aligned to xx:00 and xx:30 for the above reasoning. Then, with preSunRise = -12h and postSunSet = 12h, you'd cover the full day in 30min intervals. (Of course, the config file entries would need be a little more obvious)

Matze2 commented 1 year ago

Hi Stefan,

thanks for all the detailed explanations.

This means to achieve what I want, I just need a crontab entry

15,45 * * * *

and

force=true

in solcast_light_config.ini to skip all the optimization logic in solcast.py. No code change required.

I'll close the PR and issue since it does not give any new benefit. Thanks again for your time.

StefaE commented 1 year ago

Newest release v2.10 contains an interval = 24h setting. This is almost identical to what you achieve with force. (force allows you to pull at any interval and is mainly for debugging; the new setting calculates reasonable time intervals out of the available credits and double-checks that crontab doesn't run too often ...)

Matze2 commented 1 year ago

Thanks for your effort. Today I found time to update to the latest version. Now I am using "24h" mode. :+1:

StefaE / PVForecast

[solcast] "all-day" API request optimization #15