Comparison weather data between AC-Orsbach and EBC-station

Jun-Jiang-92 commented 1 year ago

Describe the bug I compared the data from both weather stations under the same date and time. There are a lot of differences, including:

Regarding the values from the EBC weather station:

If no data available, it uses a default value (e.g. 20 °C for ambient dry bulb temperature and 50 % for the r.h.), instead interpolating the value.
The solar radiation values (HDirNor, HDifNor and HGloHor) are all too high. Some values of HDirNor in Feburary 2023 are almost 1000 W/m² and nearly approaches the solar constant . There should be some bugs here.

Regarding the values from the AC-Orsbach:

No values in winSpe and winDir

To Reproduce Download both data in *.mos format and import with AixLib package "AixLib.BoundaryConditions.WeatherData.ReaderTMY3". Then, run the simulation.

Expected behavior The data are (almost) consistent with each other.

Screenshots grafik

Additional context None

Bug Fix Suggestion There should be some bugs by the convertion and data export.

MartinRaetz commented 11 months ago

Thank you @Jun-Jiang-92.

I see three probable causes:

There is a bug inside the tool when converting the radiation values
The measurements at EBC are not what they seem to be (e.g. "diffuse-radiation" is this really the diffuse radiation for the whole sky or just the radiation hitting the sensor when blocking direct sunlight?)
The measurements at EBC are faulty.

The interpolation with default values is currently the desired behavior, as regular interpolation does not seem much more accurate to me and I think that it is valuable to see easily when data is missing. You may argue your point of view and we can think about changing it.

Regarding the missing values, you may check directly online at the DWD (see AixWeather ReadMe: https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/) whether data for windspeed and wind direction are available for that period. If the data is available please let me know, we can have a look into the code where it gets lost.

I would appreciate if you could have a look at this. The first and most helpful thing would be finding out the exact meaning of the EBC measurements and documenting them here. Then we can check the code together, whether its correctly handled.

Jun-Jiang-92 commented 11 months ago

Alright. I think I will:

[x] Check the data sheet of EBC weather station to find out how it works for the solar radiation measurements and units of physical values
[x] Check the DWD data to confirm if there are some values for both wind parameters

Jun-Jiang-92 commented 11 months ago

@MartinRaetz I got some updates.

Regarding the EBC weather station:

There are some data sheets from the manufactor here: W:\EBC_Intern\Messtechnik\Wetterstation\Thies. The global and diffuse radiations are measured with pyranometer in W/m².
Compared with the data from aedifion, both parameters "HDifHor" and "HGloHor" in the simulation data are same as the data "global-radiation" and "diffuse-radiation" from the weather station. I noticed that, in the generated .mos data, radiations are described with the unit of "Wh/m²". And in the simulation, parameters are denoted with the unit "W/m²". I am not sure about, which kind of input does Modelica need for the simulation from the weather data and how the modelica converts the data internally. It would be better that we have a discussion about that.

Regarding the missing values of AC Orsbach:

You are right. There is not any values available under https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/10_minutes/wind/historical/, since for all of the weather stations, the historical value is only availabel to the end of 2022 or even earlier. So it means, it is not possible to achieve a complete weather data set for any of "this" year. Is that right?

Does the old weather tool have the same problem? I also checked other directories but not found if there is somewhere the wind data of last months are saved.

MartinRaetz commented 11 months ago

@Jun-Jiang-92 Thank you for that deep dive!

This is what the Tool currently expects the ERC data to be:

'4121.weatherstation.diffuse-radiation': {'core_name': 'DiffHorRad', 'time_of_meas_shift': 'foll2ind', 'unit': "Wh/m2"},
'4121.weatherstation.global-radiation': {'core_name': 'GlobHorRad', 'time_of_meas_shift': 'foll2ind', 'unit': "Wh/m2"}

Unit seems to be correct as we have hourly measurements. The data point need clarification: AixWeather expects both radiations to be only the horizontal radiations. Can you verify that the ERC measures this?

Regarding your second point "Aedifion vs. Simulation", I need clarification. Did not understand that.

Regarding the completeness of weather data: You may only pull what is available online, regardless the Tool. Old and new version use the same data origin.

Jun-Jiang-92 commented 11 months ago

@MartinRaetz The EBC weather station should measure the global horizontal irradiance (GHI) and the diffuse horizontal irradiance (DHI).

As described in the following links: https://www.hukseflux.com/applications/solar-energy-pv-system-performance-monitoring/what-is-a-pyranometer https://www.yellowhaze.in/solar-irradiance/ https://en.wikipedia.org/wiki/Solar_irradiance If the pyramometer is horizontally aligned, as on the EBC weather station, it measures the global horizontal inradiance (GHI). The GHI has a range from 0 to 1400 W/m², by which the maximum is the solar constant. Is that the expected parameter to be converted into the 'GloHorRad'?

For the comparison between Aedifion and .mos data, is the data flow now '4121.weatherstation.diffuse-radiation' -> 'DiffHorRad' -> 'HDifHor' without any conversions in the process? As results, the values in the simulation are the same as in the aedifion now. Is that true?

Now, I'm checking the conversion from DWD into the .mos data. Does the tool use "DS_10" and "GS_10" as data source now? https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/10_minutes/solar/BESCHREIBUNG_obsgermany_climate_10min_solar_de.pdf Could you show me how the conversion works for the data processing from DWD to .mos?

MartinRaetz commented 11 months ago

Hey @Jun-Jiang-92, thank you for checking and clarifying!

Perfect! In that case, the conversion of the ERC radiation is correct, as the measured value is the desired value.

And yes, DS_10 and GS_10, you find the data conversion dictionary for the DWD data here: aixweather/transformation_to_core_data/DWD.py:15

Can you, for one time step, calculate the DWD horizontal radiations and compare them with the radiations given by the Tool?

MartinRaetz commented 11 months ago

      15000;202301091000;    3;   5.4;   5.4;   0.000;-999;eor
      15000;202301091010;    3;   3.3;   3.3;   0.000;-999;eor
      15000;202301091020;    3;   2.1;   2.1;   0.000;-999;eor
      15000;202301091030;    3;   2.9;   2.9;   0.000;-999;eor
      15000;202301091040;    3;   3.2;   3.2;   0.000;-999;eor
      15000;202301091050;    3;   2.5;   2.5;   0.000;-999;eor

This is the raw DWD data, summing up the DiffHorRad for one hour (third column) results in 19.4 J/cm2. Which is divided by 3600 (Joule to Wh) and multiplied by 10000 (cm2 to m2), resulting in 53,88 Wh/m2.

As far as I can see, this is the value that the tool also provides. Please verify the correctness of the calculation. If verified, the error seems not to be within the tool but at the DWD or the ERC measurement station.

MartinRaetz commented 11 months ago

Here the raw hourly DWD data as a backup check:

15000;2023010909:42;    1;   -999;   17.0;   17.0;   0;    80.05;2023010910:00;eor
15000;2023010910:42;    1;   -999;   22.0;   22.0;   0;    75.56;2023010911:00;eor
15000;2023010911:42;    1;   -999;   30.0;   31.0;   0;    73.20;2023010912:00;eor

4th column is the DiffHorRad in J/m2. Resulting in roughly 22 J/cm2 for that hour. Resulting in 61.11 Wh/m2. Being roughly in line with the above-shown values.

Jun-Jiang-92 commented 11 months ago

@MartinRaetz Hey, Martin. I used the AixWeather WebApp again just now to export the DWD data for January 2023. Please have a look at the following data for the day of 20230109, from 0:00 to 23:00:

691200  6.28    4.57    88.72   96774.17    -0  -0  310.33  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
694800  5.95    4.41    89.84   96816.67    -0  -0  307.73  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
698400  5.78    4.07    88.81   96858.33    -0  -0  311.61  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
702000  5.42    3.43    86.94   96877.50    -0  -0  306.36  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  7.50    7.50    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
705600  5.33    3.14    85.67   96880.83    -0  -0  293.68  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  6.25    6.25    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
709200  5.66    3.38    85.17   96897.50    -0  -0  302.99  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
712800  5.93    3.50    84.32   96919.17    -0  -0  310.45  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
716400  6.16    3.50    83.09   96940.83    -0  -0  311.65  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
720000  6.26    3.71    83.69   96981.67    -0  -0  317.77  2.69    0.00    2.69    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
723600  6.32    3.92    84.61   97058.33    -0  -0  318.36  9.12    0.00    9.12    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
727200  6.39    3.77    83.38   97145.83    -0  -0  313.40  8.98    0.00    8.98    -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
730800  6.17    3.59    83.63   97210.00    -0  -0  307.59  17.18   2.77    16.39   -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
734400  5.96    3.59    84.78   97247.50    -0  -0  311.81  11.94   1.27    11.57   -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
738000  6.02    3.52    84.02   97304.17    -0  -0  316.71  7.73    0.00    7.73    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
741600  5.81    3.38    84.36   97393.33    -0  -0  311.00  2.69    0.00    2.69    -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
745200  5.56    3.26    85.10   97504.17    -0  -0  304.59  0.37    0.00    0.37    -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
748800  5.42    3.27    85.99   97610.00    -0  -0  308.90  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
752400  5.39    3.52    87.61   97706.67    -0  -0  308.68  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
756000  5.63    3.80    87.91   97816.67    -0  -0  304.21  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
759600  5.53    4.00    89.93   97937.50    -0  -0  309.99  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
763200  5.06    3.97    92.68   98065.83    -0  -0  314.03  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
766800  4.75    3.80    93.62   98190.00    -0  -0  312.18  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
770400  4.47    3.50    93.40   98301.67    -0  -0  305.46  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
774000  4.12    2.92    91.83   98415.83    -0  -0  291.00  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  5.00    5.00    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0

C9 for global horizontal radiation, C11 for diffuse horizontal radiation. The values there are not same as you calculated. It seems that the unit of values are not converted into Wh/m². Could you check that?

MartinRaetz commented 11 months ago

@Jun-Jiang-92 This should be easy to track!

Once you have installed AixWeather, you can run the example. Then you can track the made changes through "import", "transformation to core data", "core data", "transformation to .mos", "mos".

But I think the problem is most likely somewhere between "import" and "core data".

MartinRaetz commented 11 months ago

I have gone through this and found that the resampling with summing up from 10 min to hourly data does not work as intended.

This is done here: aixweather/transformation_to_core_data/DWD.py:150 https://github.com/RWTH-EBC/AixWeather/blob/daf3a8589bb89da5a6d28eccd2fff0d15bada4de/aixweather/transformation_to_core_data/DWD.py#L150C1-L157C29

I propose the following fix:

    # resample some via sum some via mean -> results in average of following hour
    df_10min = df
    df = df.resample("H").mean()  # fills nan only if all nan in interval
    for var in variables_to_sum_DWD_historical:
        if var in df.columns:
            df_sum = (
                df_10min[var].resample("H").sum(min_count=1)
            )  # fills nan only if 1 value in interval
            df[var] = df_sum

First of all, thank you for finding this bug! @Jun-Jiang-92 In order to spread the knowledge about this Tool, I would like to ask you to implement the fix. I have created a branch related to this issue, which you can checkout. After applying the changes, you will also need to adapt the "unit test" files, as the new "correct" will change. I may help you with this. Just contact me 👍

MartinRaetz commented 10 months ago

@Jun-Jiang-92, what's the status here? Let me know, if I can help you!

Jun-Jiang-92 commented 10 months ago

@MartinRaetz I suppose the bug is caused by the incorrect resample of the df. See the code below: https://github.com/RWTH-EBC/AixWeather/blob/daf3a8589bb89da5a6d28eccd2fff0d15bada4de/aixweather/transformation_to_core_data/DWD.py#L150C1-L157C29

    # resample some via sum some via mean -> results in average of following hour
    df = df.resample("H").mean()  # fills nan only if all nan in interval
    for var in variables_to_sum_DWD_historical:
        if var in df.columns:
            df_sum = (
                df[var].resample("H").sum(min_count=1)
            )  # fills nan only if 1 value in interval
            df[var] = df_sum

The df is already resampled by hour before calculating the sum of the radiation data. I will restructure the data processing here.

MartinRaetz commented 10 months ago

I have gone through this and found that the resampling with summing up from 10 min to hourly data does not work as intended.

This is done here: aixweather/transformation_to_core_data/DWD.py:150 https://github.com/RWTH-EBC/AixWeather/blob/daf3a8589bb89da5a6d28eccd2fff0d15bada4de/aixweather/transformation_to_core_data/DWD.py#L150C1-L157C29

I propose the following fix:
    # resample some via sum some via mean -> results in average of following hour
    df_10min = df
    df = df.resample("H").mean()  # fills nan only if all nan in interval
    for var in variables_to_sum_DWD_historical:
        if var in df.columns:
            df_sum = (
                df_10min[var].resample("H").sum(min_count=1)
            )  # fills nan only if 1 value in interval
            df[var] = df_sum
First of all, thank you for finding this bug! @Jun-Jiang-92 In order to spread the knowledge about this Tool, I would like to ask you to implement the fix. I have created a branch related to this issue, which you can checkout. After applying the changes, you will also need to adapt the "unit test" files, as the new "correct" will change. I may help you with this. Just contact me 👍

Yes correct, this is what I already mentioned here. Let me know, if you need help with updating the unit tests.

Update: I have written a short guide to update the unit (integration) tests, which will merge in the dev soon. You may access it here: https://github.com/RWTH-EBC/EBC-Weather-Tool/blob/c26f34b296bc48065aa5b96e8abae88b24989511/tests/info#L1

Jun-Jiang-92 commented 10 months ago

@MartinRaetz The commit 710950718bc3cec5d4ad36b69af197c063939621 fix the bug that the solor radiation values are not added up by resampling. I already checked it. Could we have a short meeting regarding the unit tests? I'm not quite sure how it works.

Jun-Jiang-92 commented 10 months ago

New comparison between the weather data from AC-Orsbach and EBC: Day 0-90: grafik Day 140-160: grafik

Now, all solar radiation values (HDirNor, HDifHor and HGloHor) from both weather stations are comparable.
All the other values keep unchanged. (Except winSpe and winDir: It seems that both default values were set to 0 by other commits, if there is no data from DWD available.)

p.s. I checked the DWD data bank again (https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/10_minutes/wind/historical/). None of the weather stations has the wind data for the year 2023 yet.

@MartinRaetz Thank you for your help!

RWTH-EBC / AixWeather

Comparison weather data between AC-Orsbach and EBC-station #27