springfall2008 / batpred

Home battery prediction and charging automation for Home Assistant, supporting many inverter types
https://springfall2008.github.io/batpred/
95 stars 32 forks source link

Cloudy days, solar clipping #886

Open slopemad opened 3 months ago

slopemad commented 3 months ago

So I've been having a think about management of solar clipping, especially on cloudy days, where I want to export as much PV as I can (because of the rate I'm paid for PV, versus the rate I can subsequently buy power at from the grid.

My setup is particularly prone to clipping, because I have a Gen1 3.6 hybrid inverter, which can charge the battery at 2600, pass through the inverter at 3600, but I have 11 x 405W panels with a theoretical peak of about 4.5kWp (but I've been seeing 4800W today, and it's not even lunchtime). So I want to make sure that I'm passing all that extra solar into the battery to avoid losing it. It's also very intermittently cloudy, so the charger could be set to, say, 1250W to soak up all that spare solar into the battery, but when the clouds come over, I'm putting too much into the battery, when I'd be better off just exporting it. (Hey, why waste >10% on thermal losses!)

So, I've started writing a little AppDaemon app, at the moment it's just reading data and not doing anything with it as I'm trying to think of the best way to handle it. It's running this every 1 minute, collecting the last 5 minutes of data regarding PV generation and inverter throughput and processing it. This is the little routine:

    def collect_sensor_data(self, kwargs=None):
        # Calculate the start and end time for the data collection (last 5 minutes)
        end_time = datetime.datetime.now()
        start_time = end_time - datetime.timedelta(minutes=5)

        # Retrieve sensor data from Home Assistant
        pvgen_entity = "sensor.givtcp_saXXXXXXXX_pv_power"
        inverter_entity = "sensor.givtcp_saXXXXXXXX_invertor_power"
        forecast_now_entity = "sensor.solcast_pv_forecast_power_now"
        forecast_next_entity = "sensor.solcast_pv_forecast_power_next_30_mins"

        pvgen_data = self.get_history_async(pvgen_entity, start_time, end_time)
        inverter_data = self.get_history_async(inverter_entity, start_time, end_time)
        pvgen_values = [int(state['state']) for states in pvgen_data for state in states]
        inverter_values = [int(state['state']) for states in inverter_data for state in states]
        forecast_now_value = self.get_state(forecast_now_entity)
        forecast_next_value = self.get_state(forecast_next_entity)

        pvgen_min = min(pvgen_values)
        pvgen_max = max(pvgen_values)
        pvgen_avg = int(statistics.mean(pvgen_values))
        pvgen_stddev = int(statistics.stdev(pvgen_values))

        inverter_min = min(inverter_values)
        inverter_max = max(inverter_values)
        inverter_avg = int(statistics.mean(inverter_values))
        inverter_stddev = int(statistics.stdev(inverter_values))

        # Process collected sensor data (example: printing)
        self.log("PV Generation Data Collected from {} to {}: {}".format(start_time, end_time, pvgen_values))
        self.log("Inverter Data Collected from {} to {}: {}".format(start_time, end_time, inverter_values))
        self.log("PV Generation Statistics - Min: {}, Max: {}, Average: {}, Std Deviation: {}".format(pvgen_min, pvgen_max, pvgen_avg, pvgen_stddev))
        self.log("Inverter Statistics - Min: {}, Max: {}, Average: {}, Std Deviation: {}".format(inverter_min, inverter_max, inverter_avg, inverter_stddev))

        self.log("Solcast Forecast now: {}".format(forecast_now_value))
        self.log("Solcast Forecast next: {}".format(forecast_next_value))

It calls the get_history_async procedure which is just cut+paste from predbat (except it looks at and end time and start time rather than days.

The idea here being that I can capture the highest solar and inverter throughput measured in the last 5 mintues, and also look at averages and standard deviations (a high standard deviation means it's cloudy).

A typical output from this would be:

2024-03-28 10:37:18.047955 INFO PVCalc:: PV Generation Data Collected from 2024-03-28 10:32:17.939639 to 2024-03-28 10:37:17.939639: [4878, 1920, 1622, 1761, 1612, 1586, 2020, 2091, 3195, 1893, 1661, 1752, 1526, 1413]
2024-03-28 10:37:18.049386 INFO PVCalc:: Inverter Data Collected from 2024-03-28 10:32:17.939639 to 2024-03-28 10:37:17.939639: [3490, 1009, 1142, 1186, 1130, 1151, 1152, 1164, 1841, 1139, 1149, 1153, 1163, 1152]
2024-03-28 10:37:18.050675 INFO PVCalc:: PV Generation Statistics - Min: 1413, Max: 4878, Average: 2066, Std Deviation: 917
2024-03-28 10:37:18.051979 INFO PVCalc:: Inverter Statistics - Min: 1009, Max: 3490, Average: 1358, Std Deviation: 642
2024-03-28 10:37:18.055044 INFO PVCalc:: Solcast Forecast now: 2794
2024-03-28 10:37:18.057885 INFO PVCalc:: Solcast Forecast next: 3392

I can see from that, the solcast forecast suggests 2794W, the peak I've seen is 4898 (because my battery charge rate was set to about 1200), and it's pretty cloudy, which can be seen by the high standard deviation figure.

My theory here being that if this little app sees maximum inverter throughput at close to maximum, then I probably want to be increasing the battery charge rate to collect more spare solar. (and similarly, if peak inverter throughput is lower than it can actually handle, then we might want to reduce throughput.

The figures derived from this could also be used to establish the cloud model used by Predbat for the next hour or so, perhaps in a way that tapers from one established here to the one calculated using the solcast forecasts?

This is very much a work in progress that I'm looking at here, but I figured I'd share it here as it may well be of interest to integrate with Predbat and get more thoughts? (I think it should be a seperate .py file which runs every minute rather than every 5 minutes, to try and capture cloudy days, minimise clipping, and maximise export).

Neomancer86 commented 1 week ago

Both, ish.

It doesn't show it in the plan as such, but here is an example of it:

image

gcoan commented 1 week ago

Have to admit I thought low power charging mode only applies with (overnight) grid charging

But the documentation doesn't explicitly say that https://springfall2008.github.io/batpred/customisation/#inverter-control-options

Worth doing a test. @Neomancer86 what does number.givtcp_xxxx_battery_charge_rate look like during the morning charging? This is the entity that controls the charging rate

Does this affect both charging from the grid and from solar? Or just from the grid? I ask because on comparing plans for tomorrow (first forecast sunny day for a few days) with it switched on/off I can see no difference whatsoever between them.

@BuhJuhWuh I would have thought the plan would be different as the battery would reach 100% SoC slower if the switch was turned on and Predbat was reducing the charging rate

Neomancer86 commented 1 week ago

Worth doing a test. @Neomancer86 what does number.givtcp_xxxx_battery_charge_rate look like during the morning charging? This is the entity that controls the charging rate

image

I will say this is not perfect and not reliable as I do have occasional clipping, it would be nice to have a 10% buffer as I still clip a bit.

gcoan commented 1 week ago

Naah, that's just shuffling between zero charge rate and full charge rate (3.6kW) which is normal Predbat behaviour ☹️.

I was expecting (hoping) it would be a charge rate somewhere in the middle, either a significantly lower figure or something that had a gradual taper or shape to it as Predbat adapted the battery charging to ensure it ended up at 100% at the end of the (solar) charge period rather than max full rate all the time.

Oh well, future enhancement needed by @springfall2008

Neomancer86 commented 1 week ago

just to note, zero charge rate is around 400W?

BuhJuhWuh commented 1 week ago

@BuhJuhWuh I would have thought the plan would be different as the battery would reach 100% SoC slower if the switch was turned on and Predbat was reducing the charging rate

Its plan for tomorrow involved no grid charging at all, only solar, which I think fits with your thought here:

Have to admit I thought low power charging mode only applies with (overnight) grid charging

So yes, future enhancement required to make the charge rate variable, along with the clipping threshold stuff!

PatrickJanssens commented 1 week ago

As we had a rather sunny day yesterday thought it might be useful to show how my system coped with clipping. As mentioned earlier I'm on the Intelligent Octopus Go tarrif, exporting at a fixed rate of 15p. So not so much worried about when to discharge. And keen on fully charging the battery at 7.5p during the cheap hours.

I manually set predbat to "manual force discharge" at 8:50 and left my system in "manual forced discharge" all the way till 17:30 and again for a short while after that. Clipping point was reached at around 10:30. As the battery wasn't full in the early morning my battery charge was 38% when clipping point was reached. (For some reason the system discharged my battery in the morning while still charging the car. Normally I would have a fully charged battery at 5:30, so the battery charge would be about 65% at 10:30)

From 10:30 my battery started charging using the excess PV. there was a cloudy spell from 11:40 till 13:20 which allowed the battery to discharge (at maximum power) and recover some 10% battery capacity in that period. After that the battery resumed charging the excess PV which charged the battery to about 80% using the excess PV. So overall gained some 40% using excess PV which I wouldn't have recoved if I would have allowed clipping to occur.

Some notes and my conclusions from this:

So coming back to my wish list for predbat:

Alternatively, Is there any way to make the "manual forced discharge" configurable using automations?

2024-06-26

gcoan commented 1 week ago

@PatrickJanssens this is very interesting, I'm trying to get my head around how this works!

Can I start by understanding the solar array size, inverter size and battery capacity?

So you started off with the battery normally full from an overnight IOG charge (although not as full as normal for this test).

Predbat would presumably have put the battery into Eco mode and so you would be exporting excess solar (how much?) and/or charging the battery at max rate if its not full.

You set manual forced discharge in Predbat (which translates to Forced Export on the inverter) for the day. When it was sunny the battery charged on excess solar, when it was cloudy the battery discharged.

If I plug my own data for my larger array in to this strategy ....

Array is 6.2kW, Gen 1 5kW hybrid inverter, 9.5kW battery. Octopus Agile and at the moment the rates have been bad overnight so been running off battery which is usually around 50% in the morning.

Solar starts charging the battery around 5am and its generally full by 10am if I do nothing.

Solar gen usually around 4-5kW, peaking at 6kW by lunchtime.

So for all the time the solar generation exceeds the inverter AC limit this strategy captures the excess solar in the battery. For the time the excess solar is less than the inverter limit the battery is discharging. So hence need to ensure the strategy is stopped sufficiently early to enable the battery to recharge sufficiently for the evening (or for me, for the night if the agile rates are > the export rate).

It might be worth setting the battery discharge rate to a value lower than the maximum to ensure the battery doesn't discharge prematurely, although that does reduce the export rates. Also more efficient to retain the charge than discharge and recharge the battery,

My current strategy for summer days has been to set Predbat into read only and to reduce the charge rate down to 1kW. Have to set R.O. as otherwise Predbat just changes the charge rate back up again. This leaves the battery only charging slowly through the morning and its usually fully charged by early afternoon after the peak generation when inverter clipping has ceased.

Using the same generation points above, with this strategy:

I don't know which strategy is more efficient/cash positive, would need to properly model it, but intuitively:

Clearly with a cheap IOG rate it makes sense to charge the battery overnight and even though you'll incur battery discharge losses, the overnight/export rate differential is more than this.

For me my solar array is not that much bigger than my inverter size so if I followed your strategy I would tend to be discharging the battery during the day and would end up having to recharge in the evening. I would also have incurred the battery discharge/charge losses which are more significant as I don't have the cheap IOG overnight rate.

Here's my PV power and SoC graph from yesterday. PV on my big array (G) didn't actually get above 5kW so actually no clipping. Battery full by 12 which is after the peak on that array: image

Looking back, 19th June was a particularly sunny day, peaks up to 5.7kW, but what I see on days like this is that I am generating so much export power I am going way above the normal grid voltage levels and the inverters keep throttling back. DNO changed the tap down for me last year so am reluctant to ask for it to be done again as this is very much self inflicted. Why the FIT array goes negative power I have no idea, its a CT clamp on the cable so makes no sense. image

In conclusion I think this shows that there isn't necessarily any one single strategy for predbat to manage clipping. I'm still leaning to my approach being better given my equipment setup.

Only problem with mine is that battery charge rate can't at present be managed dynamically via an entity in HA so I have to put Predbat into read only. But I think I can look at the predbat code to change the way this works

You asked

Alternatively, Is there any way to make the "manual forced discharge" configurable using automations?

Yes, can do it, just need to pause between setting the control in Predbat to let it catch up with the change. Here's an automation I had running earlier when Predbat wanted to discharge at 3pm every day:

alias: "Predbat: stop 3pm discharge"
description: ""
trigger:
  - platform: time
    at: "23:00:00"
action:
  - service: select.select_option
    data:
      option: "15:00:00"
    target:
      entity_id:
        - select.predbat_manual_idle
  - delay:
      minutes: 1
  - service: select.select_option
    data:
      option: "15:30:00"
    target:
      entity_id: select.predbat_manual_idle
mode: single
PatrickJanssens commented 1 week ago

@gcoan Interesting indeed

@PatrickJanssens this is very interesting, I'm trying to get my head around how this works! Can I start by understanding the solar array size, inverter size and battery capacity?

My solar array is 7.83 kWp (18 panels 435 W each), GivEnergy 5kW hybrid inverter (gen 3) and a 9,5 kWh battery.

So you started off with the battery normally full from an overnight IOG charge (although not as full as normal for this test). Predbat would presumably have put the battery into Eco mode and so you would be exporting excess solar (how much?) and/or charging the battery at max rate if its not full.

I believe so. At the start of the day the load is taken by the battery until solar generation starts. This will first start charging the battery, and any excess is exported to the grid. On this particular day, as somehow something went wrong with the car charging (maybe Octopus triggered the charging outside of an "intelligent" slot) causing the battery to be discharged quite a bit, all solar was sent to the batteries in the early morning to top them up again. From 6:30 to 8:50 it added some 15% charge to the battery.

You set manual forced discharge in Predbat (which translates to Forced Export on the inverter) for the day. When it was sunny the battery charged on excess solar, when it was cloudy the battery discharged.

Correct, as soon as I set predbat to "manual forced discharge" the inverter is set to timed export, eco mode is switched off and the inverter will export at maximum inverter AC capacity (5kW minus the house load) either by using the solar or solar+battery. Any solar DC power in excess of the maximum inverter capacity is sent to the battery. image

If I plug my own data for my larger array in to this strategy .... Array is 6.2kW, Gen 1 5kW hybrid inverter, 9.5kW battery. Octopus Agile and at the moment the rates have been bad overnight so been running off battery which is usually around 50% in the morning. Solar starts charging the battery around 5am and its generally full by 10am if I do nothing. Solar gen usually around 4-5kW, peaking at 6kW by lunchtime. When 3kW of solar generated, inverter can discharge the battery at 2kW, leading to 5kW of export. When 4kW of solar generated, inverter can discharge the battery at 1kW, leading to 5kW of export. When 5kW of solar generated, inverter is at max AC limit so can't discharge the battery at all, so also 5kW of export. When 6kW of solar generated, 5kW AC limit applies so 5kW exported, and you're saying the 1kW of excess DC solar is charged to the battery. It seems counter-intuitive as to what happens but I believe you

That is correct. If you look at my previous graph (see below) you can see that whenever there is excess solar the battery follows the same pattern, minus the house load. I presume the missing 400 W if you add the numbers is the internal load of the inverter. (The spike at 14:30 was just me playing with settings inadvertently switching off the forced discharge) 2024-06-27

So for all the time the solar generation exceeds the inverter AC limit this strategy captures the excess solar in the battery. For the time the excess solar is less than the inverter limit the battery is discharging. So hence need to ensure the strategy is stopped sufficiently early to enable the battery to recharge sufficiently for the evening (or for me, for the night if the agile rates are > the export rate).

Yes, knowing when to switch back to normal (eco?) mode is crucial. I believe using the remaining PV energy estimate from e.g. Solcast would be a good trigger for this. I choose this quite arbitrarily for now (somewhere around 12 kWh), but this is something that could be done a lot more accurately by predbat. Actually it would make sense to also dynamically increase the battery minimum SOC setting in function of how much buffer is required for clipping (reducing over time and zero once no more clipping is expected in the day) and how much charge is needed to cover evening and night loads. Especially if suddenly clouds appear the battery levels could otherwise suddenly drop more than desired.

It might be worth setting the battery discharge rate to a value lower than the maximum to ensure the battery doesn't discharge prematurely, although that does reduce the export rates. Also more efficient to retain the charge than discharge and recharge the battery,

I'm still not very convinced reducing the discharge rate matters, at least not in my case where the export tarrif is fixed. I'd rather restrict the discharge levels setting the minimum battery SOC.

My current strategy for summer days has been to set Predbat into read only and to reduce the charge rate down to 1kW. Have to set R.O. as otherwise Predbat just changes the charge rate back up again. This leaves the battery only charging slowly through the morning and its usually fully charged by early afternoon after the peak generation when inverter clipping has ceased.

Could this also not be achieved by controlling the maximum charge level for that period rather than the charge rate? Could take out some experimenting finding the optimal charge rates?

Using the same generation points above, with this strategy: When 3kW of solar generated, inverter can only charge the battery at 1kW, leading to 2kW of export. When 4kW of solar generated, inverter can again charge the battery at 1kW, leading to 3kW of export. When 5kW of solar generated, inverter charges battery at 1kW and exports 4kW. When 6kW of solar generated, 5kW AC limit applies so 5kW exported, and 1kW is charged into the battery still I don't know which strategy is more efficient/cash positive, would need to properly model it, but intuitively: Clearly with a cheap IOG rate it makes sense to charge the battery overnight and even though you'll incur battery discharge losses, the overnight/export rate differential is more than this.

Yes, with IOG tarrif the only concern here is really the number of cycles on the battery. But seems GivEnergy is quite confident their latest batteries can take some hammering as there is no longer a cycles limit in their guarantee.

For me my solar array is not that much bigger than my inverter size so if I followed your strategy I would tend to be discharging the battery during the day and would end up having to recharge in the evening. I would also have incurred the battery discharge/charge losses which are more significant as I don't have the cheap IOG overnight rate.

Makes sense. I also have quite a big margin on my solar panels, so I think on a really sunny day I would recover close the a full battery load FOC. (Of course not that many such days here in the North-East of the UK).

Here's my PV power and SoC graph from yesterday. PV on my big array (G) didn't actually get above 5kW so actually no clipping. Battery full by 12 which is after the peak on that array: image

Looking back, 19th June was a particularly sunny day, peaks up to 5.7kW, but what I see on days like this is that I am generating so much export power I am going way above the normal grid voltage levels and the inverters keep throttling back. DNO changed the tap down for me last year so am reluctant to ask for it to be done again as this is very much self inflicted. Why the FIT array goes negative power I have no idea, its a CT clamp on the cable so makes no sense. image

In conclusion I think this shows that there isn't necessarily any one single strategy for predbat to manage clipping. I'm still leaning to my approach being better given my equipment setup.

Yes, agree much depends on the tarrifs and solar setup.

Only problem with mine is that battery charge rate can't at present be managed dynamically via an entity in HA so I have to put Predbat into read only. But I think I can look at the predbat code to change the way this works

You asked Alternatively, Is there any way to make the "manual forced discharge" configurable using automations? Yes, can do it, just need to pause between setting the control in Predbat to let it catch up with the change. Here's an automation I had running earlier when Predbat wanted to discharge at 3pm every day:

Great, will try!

gcoan commented 1 week ago

@PatrickJanssens thanks for all the extra information, it really is surprising behaviour that the inverter will charge the battery when its in 'discharge mode', amazing!

Yes, knowing when to switch back to normal (eco?) mode is crucial. I believe using the remaining PV energy estimate from e.g. Solcast would be a good trigger for this.

One of the challenges I have (and appreciate my setup is probably not typical) is 3 different solar arrays. An older 16 panel 4kW FIT array (West facing) and then 28 further 390W panels that were installed in Jan 23 - the output of which would exceed a single inverter so I have two 5kW GivEnergy hybrid inverters (one with two East strings and the other with East and West strings). It does give me a long solar generation day.

My Solcast forecast is thus now the summation of all these panels and inverters and so I can't use the Solcast forecast to judge when clipping will occur. Its actually only the 5kW East array that I mentioned earlier that runs the risk of clipping, the others are OK.

Actually it would make sense to also dynamically increase the battery minimum SOC setting in function of how much buffer is required for clipping (reducing over time and zero once no more clipping is expected in the day) and how much charge is needed to cover evening and night loads. Especially if suddenly clouds appear the battery levels could otherwise suddenly drop more than desired.

The danger with setting the minimum SoC (battery reserve) is that if solar drops off or you get unexpected house load then the battery may reach that limit, stop discharging and you start grid importing instead.

It might be worth setting the battery discharge rate to a value lower than the maximum to ensure the battery doesn't discharge prematurely, although that does reduce the export rates. Also more efficient to retain the charge than discharge and recharge the battery,

I'm still not very convinced reducing the discharge rate matters, at least not in my case where the export tarrif is fixed. I'd rather restrict the discharge levels setting the minimum battery SOC.

Personally I keep the discharge rate at maximum so that if you turn the kettle on the battery picks up the load and you don't grid import.

My current strategy for summer days has been to set Predbat into read only and to reduce the charge rate down to 1kW. Have to set R.O. as otherwise Predbat just changes the charge rate back up again. This leaves the battery only charging slowly through the morning and its usually fully charged by early afternoon after the peak generation when inverter clipping has ceased.

Could this also not be achieved by controlling the maximum charge level for that period rather than the charge rate? Could take out some experimenting finding the optimal charge rates?

No, setting the max charge level (target soc) means that once the battery hits that charge level your inverter stops charging and the inverter solar throughput becomes limited to the AC limit. Any excess solar is clipped until you next increase the max charge level.

Reducing the battery charge rate delays the rate the battery charges up, enables the inverter to operate at AC limit + battery charge rate and thus capture (in the battery) the clipped solar (up to the battery charge rate limit).

PatrickJanssens commented 1 week ago

@gcoan Yes, see your point on setting the minimum and maximum SOC.

I have now set up two automations controlling the predbat forced manual discharge (thanks for your help). One sets the forced discharge in the morning (triggered at 5:30, after I get a Solcast update at 5:00) to forced discharge from 7:30 to 17:00 if the Solcast forecast for the day is above a set value. and one that switched off the forced discharge once the remaining today forecast drops below a certain value. I put the these set values as helpers that I can adjust for now, so will play with these in the next couple of days. (starting with a daily forecast trigger at 40 kWh and the remaiing today forecast trigger at 12 kWh). Will see how that behaves. Hopefully more sunny days to come so I can try this out properly.

BuhJuhWuh commented 1 week ago

So today was interesting...

A pretty sunny day, but with a big chunk of low Agile pricing in the afternoon (left hand side; tomorrow's looking tasty too): WhatsApp Image 2024-06-27 at 21 43 02

Predbat gave me one of those "weird on first glance, but totally makes sense when you think about it" plans - essentially "hold SoC low overnight, waiting for the cheap slots, then jump about like crazy to make the most of the brown electricity profits":

WhatsApp Image 2024-06-27 at 21 43 02 (3)

WhatsApp Image 2024-06-27 at 21 43 02 (1)

But then later on in the morning I had a wee look to see what it was doing - nice - exactly the kind of behaviour I'd like!

WhatsApp Image 2024-06-27 at 21 43 02 (2)

This appears to fit with your discovery/suggestion above @PatrickJanssens / @gcoan:

  • When 6kW of solar generated, 5kW AC limit applies so 5kW exported, and you're saying the 1kW of excess DC solar is charged to the battery. It seems counter-intuitive as to what happens but I believe you

Here's today's history. I have a 6.56kWp SE-facing array feeding a 9.5kWh battery through a 5kW inverter. So it's nice to see regions over 7kW! (Even starker on less smoothed chart - peak of 7.8kW.)

So it looks like a useful behaviour is there. I'm not totally certain which of the modes this corresponds to / which of the Predbat Table Card icons correspond to which mode, but I think it was either Freeze Charge or Freeze Discharge. Is there a way of using that to our advantage?