Solcast time after day ahead optimization not correct

[BDVGitHub]

Describe the bug I'm using Solcast for the solar forcast and filled in the paramters in te config en secret file and it worked before.

Since I'v updated to version 0.8.4 the predicted solar data is not on the right time stamp Its like the solar data is shifted 10h further than expected (see screenshot)

I'm using solcast also in the energy dashboard on HA and there it works fine

Expected behavior That the solar data is shown on the right timeslot

Screenshots log file

data after day ahaed optimization when you look at p_pv it start showing values from 18u but then the sun is almost gone

Home Assistant installation type

• Home Assistant Supervised

Your hardware

• OS: Linux
• Architecture: amd64

EMHASS installation type

• Docker Standalone

[davidusb-geek]

Are you on TZ +1h?

[BDVGitHub]

yes correct, I live in Belgium In the docker container the time is running correct and in the secret file the timezone is also set

[davidusb-geek]

Very strange. When starting the app with the docker run command try adding the env variable -e TIME_ZONE="Europe/Paris" to directly specify the TZ and see if it makes any difference.

[GeoDerp]

After trying @davidusb-geek 's recommendation. Try: -e TZ="Europe/Paris": So something line docker run ... -e TIME_ZONE="Europe/Paris" -e TZ="Europe/Paris ... This makes sure the Docker containers time zone is correct.

If the docker container time is correct anyways this wont fix the issue. But its worth a try.

[BDVGitHub]

I've removed the container en build it with docker run ... -e TIME_ZONE="Europe/Paris" -e TZ="Europe/Paris ...

the time is correct in the container see image, but still same issue

[BDVGitHub]

In the log file I found this error

2024-03-31 18:47:25,420 - web_server - INFO - Status: Optimal 2024-03-31 18:47:25,420 - web_server - INFO - Total value of the Cost function = -1.58 2024-03-31 18:47:25,423 - web_server - INFO - Solving for day: 30-3-2024 2024-03-31 18:47:25,424 - web_server - ERROR - Exception on /action/perfect-optim [POST] Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/datetimes.py", line 2423, in _infer_tz_from_endpoints inferred_tz = timezones.infer_tzinfo(start, end) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "pandas/_libs/tslibs/timezones.pyx", line 368, in pandas._libs.tslibs.timezones.infer_tzinfo AssertionError: Inputs must both have the same timezone, UTC+01:00 != UTC+02:00

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app response = self.full_dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request rv = self.handle_user_exception(e) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 119, in action_call opt_res = perfect_forecast_optim(input_data_dict, app.logger) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 207, in perfect_forecast_optim opt_res = input_data_dict['opt'].perform_perfect_forecast_optim(df_input_data, input_data_dict['days_list']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/emhass/optimization.py", line 538, in perform_perfect_forecast_optim data_tp = df_input_data.copy().loc[pd.date_range(start=day_start, end=day_end, freq=self.freq)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/indexes/datetimes.py", line 945, in date_range dtarr = DatetimeArray._generate_range( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/datetimes.py", line 422, in _generate_range tz = _infer_tz_from_endpoints(start, end, tz) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/datetimes.py", line 2426, in _infer_tz_from_endpoints raise TypeError( TypeError: Start and end cannot both be tz-aware with different timezones

And also this

2024-03-31 05:30:00,333 - web_server - INFO - Passed runtime parameters: {'load_cost_forecast': [0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115]} 2024-03-31 05:30:00,333 - web_server - INFO - >> Setting input data dict 2024-03-31 05:30:00,333 - web_server - INFO - Setting up needed data 2024-03-31 05:30:00,335 - web_server - INFO - Retrieving weather forecast data using method = solcast 2024-03-31 05:30:00,426 - web_server - INFO - Passed runtime parameters: {} 2024-03-31 05:30:00,426 - web_server - INFO - >> Setting input data dict 2024-03-31 05:30:00,426 - web_server - INFO - Setting up needed data 2024-03-31 05:30:00,427 - web_server - INFO - >> Publishing data... 2024-03-31 05:30:00,427 - web_server - INFO - Publishing data to HASS instance 2024-03-31 05:30:00,434 - web_server - INFO - Successfully posted to sensor.p_pv_forecast = 894.1 2024-03-31 05:30:00,437 - web_server - INFO - Successfully posted to sensor.p_load_forecast = 220.04 2024-03-31 05:30:00,440 - web_server - INFO - Successfully posted to sensor.p_deferrable0 = 0.0 2024-03-31 05:30:00,443 - web_server - INFO - Successfully posted to sensor.p_deferrable1 = 0.0 2024-03-31 05:30:00,446 - web_server - INFO - Successfully posted to sensor.p_deferrable2 = 750.0 2024-03-31 05:30:00,448 - web_server - INFO - Successfully posted to sensor.p_grid_forecast = 75.94 2024-03-31 05:30:00,451 - web_server - INFO - Successfully posted to sensor.total_cost_fun_value = -1.41 2024-03-31 05:30:00,453 - web_server - INFO - Successfully posted to sensor.optim_status = Optimal 2024-03-31 05:30:00,456 - web_server - INFO - Successfully posted to sensor.unit_load_cost = 0.155 2024-03-31 05:30:00,459 - web_server - INFO - Successfully posted to sensor.unit_prod_price = 0.058 2024-03-31 05:30:03,939 - web_server - INFO - Retrieving data from hass for load forecast using method = naive 2024-03-31 05:30:03,939 - web_server - INFO - Retrieve hass get data method initiated... 2024-03-31 05:30:07,574 - web_server - ERROR - Exception on /action/dayahead-optim [POST] Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app response = self.full_dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request rv = self.handle_user_exception(e) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(*view_args) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 104, in action_call input_data_dict = set_input_data_dict(config_path, str(data_path), costfun, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 100, in set_input_data_dict df_input_data_dayahead = utils.set_df_index_freq(df_input_data_dayahead) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/emhass/utils.py", line 726, in set_df_index_freq df = df.asfreq(sampling) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 10971, in asfreq return super().asfreq( ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/generic.py", line 8347, in asfreq return asfreq( ^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/resample.py", line 2232, in asfreq dti = date_range(obj.index.min(), obj.index.max(), freq=freq) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/indexes/datetimes.py", line 945, in date_range dtarr = DatetimeArray._generate_range( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/datetimes.py", line 446, in _generate_range i8values = generate_regular_range(start, end, periods, freq, unit=unit) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/_ranges.py", line 68, in generate_regular_range e = b + (iend - b) // stride stride + stride // 2 + 1

 ZeroDivisionError: integer division or modulo by zero

[davidusb-geek]

I tried but I'm currently not able to reproduce this.

[BDVGitHub]

When I change in config_emhass.yaml retrieve_hass_conf: freq: 60 # The time step to resample retrieved data from hass in minutes back to retrieve_hass_conf: freq: 30 # The time step to resample retrieved data from hass in minutes The solcast data is shown correct again, the reason why i changed the resample time from 30 to 60 is the following, I use in homeassistant an sensor to load the price per hour because its not always the same, this data is given per hour.

my question is how can I change the code below to output 30 minute data instead of 60 minute data?

• platform: template sensors: load_cost_forecast_list: value_template: >- {% if now().weekday() == 5 or now().weekday() == 6 %} {%- set load_cost = [0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115] %} {%- else -%} {%- set load_cost = [0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.115, 0.115] %} {%- endif %} {%- set values_all = namespace(all=[]) %} {% for i in range(load_cost | length) %} {%- set v = (load_cost[i] | float ) %} {%- set values_all.all = values_all.all + [ v ] %} {%- endfor %} {{ (values_all.all)[now().hour:24] + (values_all.all)[0:now().hour] }}

so that I can send 48 data points instead of 24 data points to the load cost forcast...

[davidusb-geek]

Ok I will try to reproduce this by changing the retrieve_hass_conf. It might be a bug, so I will do some tests. For your question regarding the code I don't know if I understand well, but you just need to expand your lists to have 48 elements.

For example this list with 3 elements:

list=[1,2,3]

Can be expanded to 6 elements like this:

list=[1,1,2,2,3,3]

[davidusb-geek]

Are you still having this issue?

[BDVGitHub]

Yes, even after updating to version 0.8.6

when using retrieve_hass_conf: freq: 30 # in the config file it works perfect when using retrieve_hass_conf: freq: 60 # the solcast data starts @ 14h

[BDVGitHub]

This is an part of the solcast data (yellow) with retrieve_hass_conf: freq: 30 # The solcast data start @ 6h30

This is an part of the solcast data (yellow) with retrieve_hass_conf: freq: 60 # The solcast data start @ 14h If you look to the steps off the solcast data, @ 14h the solcast value is "2W", @ 15h I expected "79W" but see "32W" because there is an time difference of 1h and not 30 minutes. Its like the data array from solcast is stil 30 minute data but the emhass data is 60 minute data.

[BDVGitHub]

An other question

In emhass I'm using retrieve_hass_conf: freq: 60 # instead of retrieve_hass_conf: freq: 30 #

This is because in Belgium in the weekend the tarrif is different than the tarrif on an weekday so I used the code bellow So that in the weekend an other data array is passed to emhass than the one on an weekday

Sensor in Home Assistant for load_cost_forecast

platform: template sensors: load_cost_forecast_list: value_template: >- {% if now().weekday() == 5 or now().weekday() == 6 %} {%- set load_cost = [0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.115] %} {%- else -%} {%- set load_cost = [0.115, 0.115, 0.115, 0.115, 0.115, 0.115, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.155, 0.115, 0.115] %} {%- endif %} {%- set values_all = namespace(all=[]) %} {% for i in range(load_cost | length) %} {%- set v = (load_cost[i] | float ) %} {%- set values_all.all = values_all.all + [ v ] %} {%- endfor %} {{ (values_all.all)[now().hour:24] + (values_all.all)[0:now().hour] }}

How do I adjust this sensor so it contains 30 minute data instead of 60 minute data? That way i can use retrieve_hass_conf: freq: 30 # in emhass instead of retrieve_hass_conf: freq: 60 # Thx

[davidusb-geek]

Ok there is probably a bug with Solcast not adapting to the new freq values. However in your original issue the dat was shifted 10h, so that's some improvement! ;-)

As for your second question I already answered before like this: Just update your lists in your template sensor. For example this list with 3 elements:

list=[1,2,3]

Can be expanded to 6 elements like this:

list=[1,1,2,2,3,3]