Day ahead optimization results all over the place

[charliesjc]

I'm not sure if this is a bug or if I've managed to configure something incorrectly. I'm using the Docker version of Emhass and my day ahead optimization results sometimes work but mostly I end up with completely impossible values and a graph that is just flat with insane spikes everywhere. I've taken a screen grab of the table to show you what I mean:

I've tried different solvers all to no avail. One weird thing is the GLPK one doesn't do anything at all if I use it and I end up with the load and pv forecast showing fine, and the battery flat lined at the target value.

The logs don't give any info.

[Octofinger]

Post your configuration and any runtime parameters to your call. There's something that makes the optimization lost. Posting the emhass logs from the call may help. Also, check the optim_status variable. I bet it says infeasible.

[charliesjc]

I managed to sort it out, I had modified certain parameters in the config yaml that the system didn't like. I'm slowly getting something workable.

I have never seen it say anything other than infeasible. I've tried tweaking the offpeak and peak times etc but it never changes anything. I also don't have peak or off peak tariffs where I live and to make it more interesting I have rolling blackouts that have a schedule I can pick up in HASS. Working out how to implement that would be quite handy. - suppose it would have to be treated almost like a peak time window but the battery would need to be charged in anticipation of the blackout window to a certain amount.

I also don't sell back to the grid because it's not worth the pittance I'd get and our utility provider insists we'd have to be net users anyway so I could sell maybe 50-100kwh and that's that.

[davidusb-geek]

I have never seen it say anything other than infeasible.

How is this possible, please share your configuration

[davidusb-geek]

I also don't have peak or off peak tariffs where I live and to make it more interesting I have rolling blackouts that have a schedule I can pick up in HASS. Working out how to implement that would be quite handy. - suppose it would have to be treated almost like a peak time window but the battery would need to be charged in anticipation of the blackout window to a certain amount.

Yes this can be treated by setting some extremely expensive grid energy prices during your scheduled blackout period. This way the system should converge to a solution where your battery was prepared to maximize self-consumption during that period. But solve your feasibility issue first.

[charliesjc]

I have never seen it say anything other than infeasible.

How is this possible, please share your configuration

Here it is. I wonder if it doesn't have something to do with the inverter selection? I chose the closest thing to what I actually have, which is a Sunsynk 8kW unit. There are no Deye/Sunsynk/Fusion inverters anywhere on the list you're using.

# Configuration file for EMHASS

retrieve_hass_conf:
  freq: 30 # The time step to resample retrieved data from hass in minutes
  days_to_retrieve: 4 # We will retrieve data from now and up to days_to_retrieve days
  var_PV: 'sensor.deyeinvertermaster_pv_power' # Photovoltaic produced power sensor in Watts
  var_load: 'sensor.inverter_load_w_o_deferrable_load' # Household power consumption sensor in Watts (deferrable loads should be substracted)
  load_negative: False # Set to True if the retrived load variable is negative by convention
  set_zero_min: True # A special treatment for a minimum value saturation to zero. Values below zero are replaced by nans
  var_replace_zero: # A list of retrived variables that we would want  to replace nans with zeros
  - 'sensor.deyeinvertermaster_pv_power'
  var_interp: # A list of retrived variables that we would want to interpolate nan values using linear interpolation
  - 'sensor.deyeinvertermaster_pv_power'
  - 'sensor.inverter_load_w_o_deferrable_load'
  method_ts_round: 'nearest' # Set the method for timestamp rounding, options are: first, last and nearest

optim_conf:
  set_use_battery: True # consider a battery storage
  delta_forecast: 1 # days
  num_def_loads: 1
  P_deferrable_nom: # Watts
  - 2000.0
  def_total_hours: # hours
  - 2
  def_start_timestep: # timesteps
  - 12
  def_end_timestep: # timesteps
  - 18
  treat_def_as_semi_cont: # treat this variable as semi continuous 
  - True
  set_def_constant: # set as a constant fixed value variable with just one startup for each 24h
  - False
  weather_forecast_method: 'scrapper' # options are 'scrapper', 'csv', 'list', 'solcast' and 'solar.forecast'
  load_forecast_method: 'naive' # options are 'csv' to load a custom load forecast from a CSV file or 'naive' for a persistance model
  load_cost_forecast_method: 'hp_hc_periods' # options are 'hp_hc_periods' for peak and non-peak hours contracts and 'csv' to load custom cost from CSV file 
  list_hp_periods: # list of different tariff periods (only needed if load_cost_forecast_method='hp_hc_periods')
  - period_hp_1:
    - start: '02:30'
    - end: '15:00'
  - period_hp_2:
    - start: '17:00'
    - end: '20:00'
  load_cost_hp: 0.292 # peak hours load cost in €/kWh (only needed if load_cost_forecast_method='hp_hc_periods')
  load_cost_hc: 0.141 # non-peak hours load cost in €/kWh (only needed if load_cost_forecast_method='hp_hc_periods')
  prod_price_forecast_method: 'constant' # options are 'constant' for constant fixed value or 'csv' to load custom price forecast from a CSV file
  prod_sell_price: 0.043 # power production selling price in €/kWh (only needed if prod_price_forecast_method='constant')
  set_total_pv_sell: False # consider that all PV power is injected to the grid (self-consumption with total sell)
  lp_solver: 'default' # set the name of the linear programming solver that will be used. Options are 'PULP_CBC_CMD', 'GLPK_CMD' and 'COIN_CMD'. 
  lp_solver_path: 'empty' # set the path to the LP solver, COIN_CMD default is /usr/bin/cbc
  set_nocharge_from_grid: False # avoid battery charging from the grid
  set_nodischarge_to_grid: True # avoid battery discharging to the grid
  set_battery_dynamic: False # add a constraint to limit the dynamic of the battery power in power per time unit
  battery_dynamic_max: 0.9 # maximum dynamic positive power variation in percentage of battery maximum power
  battery_dynamic_min: -0.9 # minimum dynamic negative power variation in percentage of battery maximum power
  weight_battery_discharge: 0.1 # weight applied in cost function to battery usage for discharge
  weight_battery_charge: 0.1 # weight applied in cost function to battery usage for charge

plant_conf:
  P_from_grid_max: 9000 # The maximum power that can be supplied by the utility grid in Watts
  P_to_grid_max: 9000 # The maximum power that can be supplied to the utility grid in Watts
  module_model: # The PV module model
  - 'CSI_Solar_Co__Ltd__CS6W_545MB'
  inverter_model: # The PV inverter model
  - 'Ginlong_Technologies_Co___Ltd___RHI_1P8K_HVES_5G__240V_'
  surface_tilt: # The tilt angle of your solar panels
  - 15
  surface_azimuth: # The azimuth angle of your PV installation
  - 0
  modules_per_string: # The number of modules per string
  - 10 
  strings_per_inverter: # The number of used strings per inverter
  - 1
  Pd_max: 8000 # If your system has a battery (set_use_battery=True), the maximum discharge power in Watts
  Pc_max: 4800 # If your system has a battery (set_use_battery=True), the maximum charge power in Watts
  eta_disch: 0.95 # If your system has a battery (set_use_battery=True), the discharge efficiency
  eta_ch: 0.95 # If your system has a battery (set_use_battery=True), the charge efficiency
  Enom: 12000 # If your system has a battery (set_use_battery=True), the total capacity of the battery stack in Wh
  SOCmin: 0.4 # If your system has a battery (set_use_battery=True), the minimun allowable battery state of charge
  SOCmax: 1.0 # If your system has a battery (set_use_battery=True), the minimun allowable battery state of charge
  SOCtarget: 0.6 # If your system has a battery (set_use_battery=True), the desired battery state of charge at the end of each optimization cycle

[davidusb-geek]

Try changing def_start_timestep and def_end_timestep to 0 for now. Also try setting the default 0.0 for weight_battery_discharge and weight_battery_charge. Your azimuth got to be wrong. That 0 means that your panels are facing north. The azimuth convention is defined as degrees east of north (e.g. North=0, South=180, East=90, West=270). Typically PV panels are installed facing south. Also you only have 10 modules for a 8kW inverter? What is the power of your modules? Yes you can test different inverters from that list, the most important thing is just to find a similar nominal power as yours.

Please share your shell or REST command used to launch the optimization.

[charliesjc]

Thanks I'll give those settings a try.

The azimuth is spot on: I'm in the Southern Hemisphere (South Africa), and all my panels face true north. I actually have 12x 545W panels but because I get such bad shading here I chose 10 to get closer to my real maximum at this time of the year. I also have them set up as 2 strings of 6 but while I was troubleshooting the initial problems with the calculations I just set it to 1 string in the config. I will be pushing a PR soon to add the damping setting for the forecast.solar integration to try and deal with the early morning/late evening shading.

As for the weight_battery_discharge and weight_battery_charge I only changed those this morning to see if anything different would happen.

For my testing I click the button on the webpage but otherwise I've been launching the day-optim rest api from hass using this (as per your quickstart guide): curl -i -H \"Content-Type:application/json\" -X POST -d '{}' http://localhost:5000/action/dayahead-optim

[davidusb-geek]

Oh yes sothern hemisphere, my bad. Ok and so when you launch that with the logger set at INFO what does it says?

[charliesjc]

Here are the logs (I changed the weight back to 0.0 and set the timestep to 0 and 0.

2024-05-14 08:35:16,641 - web_server - INFO - Launching the emhass webserver at: http://0.0.0.0:5000
2024-05-14 08:35:16,641 - web_server - INFO - Home Assistant data fetch will be performed using url: http://192.168.10.9:8123/
2024-05-14 08:35:16,641 - web_server - INFO - The data path is: /app/data
2024-05-14 08:35:16,642 - web_server - INFO - Using core emhass version: 0.9.1
waitress   INFO  Serving on http://0.0.0.0:5000
2024-05-14 08:35:19,904 - web_server - INFO - EMHASS server online, serving index.html...
2024-05-14 08:35:23,889 - web_server - INFO - Passed runtime parameters: {}
2024-05-14 08:35:23,890 - web_server - INFO -  >> Setting input data dict
2024-05-14 08:35:23,890 - web_server - INFO - Setting up needed data
2024-05-14 08:35:23,908 - web_server - INFO - Retrieving weather forecast data using method = scrapper
2024-05-14 08:35:25,498 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-05-14 08:35:25,498 - web_server - INFO - Retrieve hass get data method initiated...
2024-05-14 08:35:31,182 - web_server - INFO -  >> Performing dayahead optimization...
2024-05-14 08:35:31,182 - web_server - INFO - Performing day-ahead forecast optimization
2024-05-14 08:35:31,183 - web_server - INFO - Perform optimization for the day-ahead
2024-05-14 08:35:31,187 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> 0
2024-05-14 08:35:31,187 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> 0
2024-05-14 08:35:31,206 - web_server - WARNING - Solver default unknown, using default
Welcome to the CBC MILP Solver 
2024-05-14 08:35:31,237 - web_server - INFO - Status: Infeasible
2024-05-14 08:35:31,237 - web_server - INFO - Total value of the Cost function = -0.42
Version: 2.10.3 
Build Date: Dec 15 2019 
command line - /usr/local/lib/python3.11/dist-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/091c9460a70f424ba6ba9d0a762904e0-pulp.mps -max -timeMode elapsed -branch -printingOptions all -solution /tmp/091c9460a70f424ba6ba9d0a762904e0-pulp.sol (default strategy 1)
At line 2 NAME          MODEL
At line 3 ROWS
At line 487 COLUMNS
At line 6392 RHS
At line 6875 BOUNDS
At line 7356 ENDATA
Problem MODEL has 482 rows, 384 columns and 5520 elements
Coin0008I MODEL read with 0 errors
Option for timeMode changed from cpu to elapsed
Problem is infeasible - 0.00 seconds
Option for printingOptions changed from normal to all
Total time (CPU seconds):       0.01   (Wallclock seconds):       0.01
2024-05-14 08:35:31,510 - web_server - INFO -  >> Sending rendered template table data

[davidusb-geek]

Ok I don't see anything wrong with your configuration to fall on that unfeasible condition. Probably try with a different solver setting. Set for example: lp_solver: 'COIN_CMD' and lp_solver_path: '/usr/bin/cbc'. Just to test. If else then this is a problem with your data on sensors: sensor.deyeinvertermaster_pv_power and sensor.inverter_load_w_o_deferrable_load. What those looks like?

One quick test that we can do is to pass some dummy data to the optimizer that overrides those two sensors. To do that you can use this:

curl -i -H \"Content-Type:application/json\" -X POST -d '{\"pv_power_forecast\":[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 350, 706.1, 1230.9, 2567.5, 3766.35, 5249.45, 8989.65, 8486.5, 15394.65, 5821.65, 5233.40, 7795.5, 10456.30, 7783.8, 5833.65, 7583.15, 6955.65, 8600.65, 4103.75, 4022.04, 1258.15, 396.25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \"load_power_forecast\":[3410.0, 382.0, 2099.0, 2790.0, 3931.0, 515.0, 1943.0, 1996.0, 4003.0, 2850.0, 1906.0, 2973.0, 1145.0, 533.0, 3737.0, 3548.0, 404.0, 3215.0, 3260.0, 2255.0, 3968.0, 2473.0, 1764.0, 3693.0, 1982.0, 1328.0, 1211.0, 109.0, 3882.0, 3909.0, 3528.0, 1632.0, 1074.0, 2803.0, 467.0, 992.0, 4222.0, 487.0, 2154.0, 862.0, 2924.0, 1732.0, 3834.0, 3795.0, 3778.0, 635.0, 2110.0, 2150.0]}' http://localhost:5000/action/dayahead-optim

Here we are explicitly passing pv_power_forecast for the PV data and load_power_forecast for the load power data. At least with this we can check that the solver and the optimization works and that there is no other problem with your configuration parameters.

[charliesjc]

Your command line overrides work and I see Optimal at the bottom. It must be my sensors. Perhaps the format/type of data they are capturing is not correct. Here is the response from the api call to HASS for the load:

[
    [
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "993",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:10+00:00",
            "last_reported": "2024-05-14T10:11:10+00:00",
            "last_updated": "2024-05-14T10:11:10+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        },
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "1002",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:20.399422+00:00",
            "last_reported": "2024-05-14T10:11:20.399422+00:00",
            "last_updated": "2024-05-14T10:11:20.399422+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        },
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "966",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:24.359031+00:00",
            "last_reported": "2024-05-14T10:11:24.359031+00:00",
            "last_updated": "2024-05-14T10:11:24.359031+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        },
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "1010",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:28.403567+00:00",
            "last_reported": "2024-05-14T10:11:28.403567+00:00",
            "last_updated": "2024-05-14T10:11:28.403567+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        },
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "1008",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:32.514132+00:00",
            "last_reported": "2024-05-14T10:11:32.514132+00:00",
            "last_updated": "2024-05-14T10:11:32.514132+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        },
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "977",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:36.380832+00:00",
            "last_reported": "2024-05-14T10:11:36.380832+00:00",
            "last_updated": "2024-05-14T10:11:36.380832+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        }
    ]
]

[davidusb-geek]

I don't see anything wrong there. But what do they look like directly in HA?

[charliesjc]

The data comes through every 2-3 seconds from my ESPHome sensor connected directly to the inverter. Does the sensor need to have fewer data points and log in Wh instead of W?

[charliesjc]

Okay I'm making some progress:

If I manually send in the load forecast using my data (I just used the values it gave me on the webpage) then it is still infeasible. If I take the PV data from the webpage and send it in manually then it shows as Optimal. I'm guessing the data coming from "scrapper" is not working too well for me. I'm going to try Solcast and Forecast.Solar and see what happens.

[charliesjc]

Confirmed. Using it the way I was with Solcast and Forecast.Solar seems to be fine. It shows Optimal every time.

[davidusb-geek]

Ok. Then use those methods instead. But I'm still intrigued by why the scrapper method does not work for you. Probably no data available on your lan/lon? That is a methods that have been working perfectly well since long time.

[charliesjc]

Ok. Then use those methods instead. But I'm still intrigued by why the scrapper method does not work for you. Probably no data available on your lan/lon? That is a methods that have been working perfectly well since long time.

I'm also confused by it. But the scrapper data is there, and if you copy the values as they're displayed on the webpage under P_PV and use them via the command line then it works.

Like I mentioned in my earlier post I'll end up using forecast.solar and customize it slightly because it allows me to use a custom horizon so I can get more realistic forecasts based on the shadowing around me.

[davidusb-geek]

Ok no problem. Please close this issue if the original problem is solved. Feel free to open a new issue if needed