Time limit for deferrable loads

michaelpiron commented 8 months ago

This pull request introduces a new parameter, allowing the user to specify a time limit for each deferrable load (in MPC optimization).

The new parameter in the optimizer, called def_end_timestep, is defined as "The timestep before which each deferrable load should operate." The optimizer will get an extra constraint stating that the deferrable load should not consume any energy after the specified timestep. If a value of 0 (or negative) is provided, no extra constraint is added.

Example: "def_end_timestep": [3,0] This is the case of two deferrable loads, whereby the first load should operate in the first three timesteps of the optimization window. The second load can use the complete optimization window (cf the value of 0).

There's also a check that the available timeframe until def_end_timestep is > the timesteps needed to run the load for the specified operating hours (def_total_hours). If not, the def_end_timestep is updated automatically to respect that constraint.

davidusb-geek commented 8 months ago

Thanks @michaelpiron for this work. Some questions: Why did you limited this to MPC? Could we extend it to the day ahead optimization? This can also be really useful for those types of full-day optimization. To extend to a real window with a start and an end period, can we proceed as you did but now we have to add a def_start_timestep parameter?

michaelpiron commented 8 months ago

Automated tests seem to be failing on a piece of code I didn't change:

======================================================================
FAIL: test_yaml_parse_wab_server (tests.test_retrieve_hass.TestRetrieveHass)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/emhass/emhass/tests/test_retrieve_hass.py", line 78, in test_yaml_parse_wab_server
    self.assertTrue(list(params['optim_conf'][7].keys())[0] == 'weather_forecast_method')
AssertionError: False is not true

----------------------------------------------------------------------

Didn't change anything in the retrieve_hass domain

michaelpiron commented 8 months ago

Thanks @michaelpiron for this work. Some questions: Why did you limited this to MPC? Could we extend it to the day ahead optimization? This can also be really useful for those types of full-day optimization. To extend to a real window with a start and an end period, can we proceed as you did but now we have to add a def_start_timestep parameter?

We could extend it to day-ahead as well, I can look into that.

Including a real window with a specified start and end would indeed be a nice extra addition. I'd need to include some extra checks Example:

start timestep cannot be higher than end timestep.
the number of timesteps between start and end should be at least enough to meet the constraint of the specified operating hours.

If these exceptions occur, I would let the system automatically override the specified start and end timesteps, to increase the chances of reaching a successful optimization.

Before starting a second iteration (with day-ahead and start timestep), do you agree with the way I built this new feature?

davidusb-geek commented 8 months ago

Automated tests seem to be failing on a piece of code I didn't change:

======================================================================
FAIL: test_yaml_parse_wab_server (tests.test_retrieve_hass.TestRetrieveHass)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/emhass/emhass/tests/test_retrieve_hass.py", line 78, in test_yaml_parse_wab_server
    self.assertTrue(list(params['optim_conf'][7].keys())[0] == 'weather_forecast_method')
AssertionError: False is not true

----------------------------------------------------------------------

Didn't change anything in the retrieve_hass domain

Yes this is tricky and not optimal, I know. You just added a new parameter to the configuration, so as you can see in the code we are using the index of the parameters list to access this data which is not optimal. So you need to increase the indexes of parameters that are after you new parameters def_end_timestep.

So for example that line of code that is failing should be changed to

self.assertTrue(list(params['optim_conf'][8].keys())[0] == 'weather_forecast_method')

And there are many more like this in the code.

This is an old bad coding behavior that was identified long time ago by @smurfix and that is actually being fixed right now by @GeoDerp in #149

The correct thing to do is to access that data by the dictionary key and not by the list index. But before been able to do that, the list configurations should be changed to dicts. Hence the work on #149

michaelpiron commented 8 months ago

That's strange: I used a new index 25 for the new parameter, not changing the old indices (in order not to break anything). I assume the index change of load_forecast_method from 7 to 8 dates from an earlier change? Is it OK to keep the index 25, or should I shift them all?

What I currently did in web_server.py:

def build_params(params, options, addon):
    if addon == 1:
        # Updating variables in retrieve_hass_conf

        # Updating variables in optim_conf
        params['optim_conf'][0]['set_use_battery'] = options['set_use_battery']
        params['optim_conf'][2]['num_def_loads'] = options['number_of_deferrable_loads']
        params['optim_conf'][3]['P_deferrable_nom'] = [i['nominal_power_of_deferrable_loads'] for i in options['list_nominal_power_of_deferrable_loads']]
        params['optim_conf'][4]['def_total_hours'] = [i['operating_hours_of_each_deferrable_load'] for i in options['list_operating_hours_of_each_deferrable_load']]
        params['optim_conf'][25]['def_end_timestep'] = [i['end_timesteps_of_each_deferrable_load'] for i in options['list_end_timesteps_of_each_deferrable_load']]
        params['optim_conf'][5]['treat_def_as_semi_cont'] = [i['treat_deferrable_load_as_semi_cont'] for i in options['list_treat_deferrable_load_as_semi_cont']]
        params['optim_conf'][6]['set_def_constant'] = [False for i in range(len(params['optim_conf'][3]['P_deferrable_nom']))]
        params['optim_conf'][8]['load_forecast_method'] = options['load_forecast_method']
        start_hours_list = [i['peak_hours_periods_start_hours'] for i in options['list_peak_hours_periods_start_hours']]
        end_hours_list = [i['peak_hours_periods_end_hours'] for i in options['list_peak_hours_periods_end_hours']]
        num_peak_hours = len(start_hours_list)
        list_hp_periods_list = [{'period_hp_'+str(i+1):[{'start':start_hours_list[i]},{'end':end_hours_list[i]}]} for i in range(num_peak_hours)]
        params['optim_conf'][10]['list_hp_periods'] = list_hp_periods_list
    ..........
        params['optim_conf'][23]['weight_battery_discharge'] = options['weight_battery_discharge']
        params['optim_conf'][24]['weight_battery_charge'] = options['weight_battery_charge']
        # Updating variables in plant_conf

davidusb-geek commented 8 months ago

Unfortunately you need to shift all of them. It is tedious, I know :-(

michaelpiron commented 8 months ago

I have added the def_start_timestep as well, and improved the situation about the dict indices, inspired by #149 . However, I'm stuck on a remaining error:

File "/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/emhass-0.6.2-py3.9.egg/emhass/utils.py", line 410, in <genexpr>
    retrieve_hass_conf = dict((key,d[key]) for d in input_conf['retrieve_hass_conf'] for key in d)
TypeError: string indices must be integers

I will have a further look into it, but any feedbacks are very welcome

codecov[bot] commented 8 months ago

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (5691360) 89.52% compared to head (04dae29) 89.56%. Report is 5 commits behind head on master.

Files	Patch %	Lines
src/emhass/optimization.py	82.75%	5 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #153 +/- ## ========================================== + Coverage 89.52% 89.56% +0.03% ========================================== Files 6 6 Lines 1442 1485 +43 ========================================== + Hits 1291 1330 +39 - Misses 151 155 +4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

michaelpiron commented 8 months ago

OK, automated tests have passed. I will continue doing functional testing, therefore changed it into a draft PR

michaelpiron commented 8 months ago

Edge cases are working as expected:

Edge case: start > end

davidusb-geek commented 8 months ago

Ok so this will catch incoherent time windows entered by the user? If the proposed window is 0 --> 0 it means that no windows is defined for that deferrable load?

michaelpiron commented 8 months ago

Ok so this will catch incoherent time windows entered by the user? If the proposed window is 0 --> 0 it means that no windows is defined for that deferrable load?

Correct. I only enforce an extra constraint if the specified def_start_timestep & def_end_timestep are valid, making chances of a successful optimization as high as possible.

More concretely:

Start = 0 means: start of the prediction horizon window. No constraint added
End = 0 means: end of the prediction horizon window. No constraint added
If user specifies a negative value for start or end, the value will be changed to 0. No constraint added
If start timestep > end timestep the optimizer will not take it into account. (Both start & end timestep are changed to 0). No constraint added
If the number of timesteps between start & end is too short to meet the def_total_hours constraint, a warning message will be shown in the log, warning the user that the optimizer won't find an feasible solution. Constraints for start & end timestep are enforced and optimization continues. I tried to find a logic to automatically make the defined optimization window longer, but there's too many degrees of freedom (put the start earlier? put the end later? a bit of both?)

michaelpiron commented 8 months ago

Happy flow

Example of a valid case: The car needs to charge 2h at nominal power. I specify a timewindow of 7 timesteps = 3.5h (between 17h30 & 21h)

Forcing the deferrable load 1 to operate between 17h30 and 21h:

Outcome:

michaelpiron commented 8 months ago

Edge case: timewindow too short

Example case where the defined timewindow for deferrable load 1 is too short to meet the def_total_hours constraint: The car needed to charge 2h at nominal power. I specified a timewindow of 1 timestep = 0.5h:

Result:

michaelpiron commented 8 months ago

Day-ahead optimization

Day ahead optimization also works with the timewindow constraints.

However, DA optimization doesn't take the timewindow constraints as runtime parameters (just like def_total_hours for example), but rather takes the constraints from the add-on configuration screen:

Log of the optimization:

Result:

michaelpiron commented 8 months ago

Edge case: end optimization window deferrable load > prediction horizon

If the user specified a timewindow that goes beyond the prediction horizon, the timewindow is adjusted automatically. In the case below, the prediction horizon has 26 timesteps.

michaelpiron commented 8 months ago

Code is ready for review

davidusb-geek commented 8 months ago

Day-ahead optimization

Day ahead optimization also works with the timewindow constraints.

However, DA optimization doesn't take the timewindow constraints as runtime parameters (just like def_total_hours for example), but rather takes the constraints from the add-on configuration screen

In this specific example for day-ahead optimization no constraint was added to the LP problem right?

michaelpiron commented 8 months ago

Day-ahead optimization

Day ahead optimization also works with the timewindow constraints. However, DA optimization doesn't take the timewindow constraints as runtime parameters (just like def_total_hours for example), but rather takes the constraints from the add-on configuration screen

In this specific example for day-ahead optimization no constraint was added to the LP problem right?

There was: deferrable load 1 was only allowed to operate after the first timestep.

davidusb-geek commented 8 months ago

There was: deferrable load 1 was only allowed to operate after the first timestep.

Yes but then what is the meaning of fixing end-timestep to zero? that's the bit I don't get

michaelpiron commented 8 months ago

There was: deferrable load 1 was only allowed to operate after the first timestep.

Yes but then what is the meaning of fixing end-timestep to zero? that's the bit I don't get

The zero for def_end_timestep means that we don't impose an end limit, so the endpoint equals the end of the prediction horizon.

I've visualized some cases: Yellow is the prediction horizon of the optimization. The blue bars are the deferrable load timewindows that correspond to the provided [start, end] timesteps:

michaelpiron commented 8 months ago

Day-ahead optimization

Here's another example of DA optimization with specified timewindow for deferrable load 1: EMHASS config: Deferrable load only allowed between timestep 3 & 9

davidusb-geek commented 8 months ago

Ok, understood. Thanks for the clarifications.

davidusb-geek commented 8 months ago

There was: deferrable load 1 was only allowed to operate after the first timestep.

Yes but then what is the meaning of fixing end-timestep to zero? that's the bit I don't get

The zero for def_end_timestep means that we don't impose an end limit, so the endpoint equals the end of the prediction horizon.

I've visualized some cases: Yellow is the prediction horizon of the optimization. The blue bars are the deferrable load timewindows that correspond to the provided [start, end] timesteps:

It may be a good idea to put this image somewhere in the documentation. We can add a section about using this new feature in the Example configurations page. What do you think?

michaelpiron commented 8 months ago

Hi David, I was thinking the same. Let me see if I can find some time to do this.

michaelpiron commented 8 months ago

@davidusb-geek will have some time on Tuesday to work on documentation. Will make a new pull request for that

davidusb-geek / emhass