Missing wind speed and wind direction by pulling DWD historical data

Jun-Jiang-92 commented 3 days ago

Describe the bug By pulling the DWD historical data, there is no data for wind speed and wind direction.

To Reproduce Steps to reproduce the behavior:

Go to 'examples/e1_pull_DWD_historical_to_all_output_formats.py'

Set the DWD-pull project as:

DWD_pull_project = ProjectClassDWDHistorical(
    start=dt.datetime(2023, 1, 1),  # Pull the data of January 2023
    end=dt.datetime(2023, 2, 1),
    station=15000,  # Set the weather station as 15000: Aachen-Orsbach
    # specify whether nan-values should be filled when exporting
    fillna=True,
    # define results path if desired
    abs_result_folder_path=None,
)

Check the values of wind speed and wind direction in the pulled data

Expected behavior Pulled data include wind speed and direction

Screenshots

#1:for TMY3reader
double tab1(745,30)
#LOCATION,Aachen-Orsbach,,,,,50.798,6.024,0,something
#Explanation of Location line:
#   Element 7: latitude
#   Element 8: longitude
#   Element 9: time zone in hours from UTC
#
#DATA PERIODS, data available from 2023-01-01 00:00:00 (second=0) to 2023-02-01 00:00:00 (second=2678400)
# info: TMY3Reader requirement: Time 0 = 01.01. 00:00:00 at local time (see time zone above)
#
#USED DATA-COLLECTOR: ebc-weather-tool with input source DWD Historical (collected at 2024-07-05 11:33:27.679670) "-0" marks not available data
#Info: The AixLib/IBPSA TMY3reader requires the below mentioned units and measurement times. Last Check: 28.02.2022
#C1 Time in seconds. Beginning of a year is 0s.
...
#C16 Wind direction at indicated time. N=0, E=90, S=180, W=270
#C17 Wind speed in m/s at indicated time
#C18 Total sky cover in tenth at indicated time
...
#C30 Liquid precipitation quantity, TMY3Reader: not used
0   14.62   4.76    51.62   98364.17    -0  -0  347.88  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  6.25    6.25    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
3600    14.23   4.17    50.81   98398.33    -0  -0  345.08  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
7200    14.07   4.38    52.04   98424.17    -0  -0  356.32  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
10800   13.88   4.93    54.81   98453.33    -0  -0  355.89  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
14400   13.56   5.49    58.20   98483.33    -0  -0  355.29  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  10.00   10.00   -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0
18000   13.29   6.14    61.97   98525.83    -0  -0  348.69  0.00    0.00    0.00    -0  -0  -0  -0  -0  -0  8.75    8.75    -0  20000.00    -0  -0  -0  -0  -0  -0  -0  -0  -0

Bug Fix Suggestion There are two data sources of wind speed in the CDC:

Both sources include the wind data of January 2023, e.g.:

From https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/10_minutes/wind/historical/10minutenwerte_wind_15000_20200101_20231231_hist.zip

STATIONS_ID;MESS_DATUM;  QN;FF_10;DD_10;eor
...
  15000;202301010000;    3;  12.3; 220;eor
  15000;202301010010;    3;  11.0; 220;eor
  15000;202301010020;    3;  10.1; 220;eor
  15000;202301010030;    3;  10.0; 220;eor
  15000;202301010040;    3;  10.6; 220;eor
  15000;202301010050;    3;  10.8; 220;eor
  15000;202301010100;    3;  12.3; 220;eor

From https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/hourly/wind/historical/stundenwerte_FF_15000_20110401_20231231_hist.zip

STATIONS_ID;MESS_DATUM;QN_3;   F;   D;eor
...
  15000;2023010100;   10;  12.0; 220;eor
  15000;2023010101;   10;  10.8; 220;eor
  15000;2023010102;   10;  12.2; 220;eor
  15000;2023010103;   10;  12.0; 220;eor
  15000;2023010104;   10;  11.2; 220;eor
  15000;2023010105;   10;  11.2; 220;eor
  15000;2023010106;   10;  10.7; 220;eor

For the wind data, now the code will try to pull the data from hourly data bank. I changed the source to 10-minute data bank and the wind data appeared.

However, the the tool should also work with hourly databank, because the data also exist there. I guess there is a bug by pulling the data or by processing the data pulled.

MartinRaetz commented 3 days ago

@Jun-Jiang-92 that looks like an important improvement, thank you!

To help you get started:

Here is defined which data is pulled from hourly or 10min Sources of DWD: https://github.com/RWTH-EBC/EBC-Weather-Tool/blob/9c8bad94e33196fa7723113caafe8fef2abfe293/aixweather/imports/DWD.py#L59

You may addtionally check here if the data point name is correct e.g. "wind_direction" and make sure the "time_of_meas_shift" is correct! For 10min data it would be "prec2ind" due to the resampling methodology. https://github.com/RWTH-EBC/EBC-Weather-Tool/blob/9c8bad94e33196fa7723113caafe8fef2abfe293/aixweather/transformation_to_core_data/DWD.py#L112

In theory, the rest should be handled by aixweathers logic. If not, there might be a bug.

If you get stuck, get in contact! Otherwise, I will be happy to review your PR :)

Jun-Jiang-92 commented 3 days ago

The reason is that DWD uses two different names for these two data resouces of wind data. The mapping in the program now is actually for the source of 10-min data. https://github.com/RWTH-EBC/AixWeather/blob/b6adf79b33aa35f4b29e235e8f09010e77608cdd/aixweather/transformation_to_core_data/DWD.py#L39

The hourly wind data are named differently. As also mentioned in the programm as comment: https://github.com/RWTH-EBC/AixWeather/blob/b6adf79b33aa35f4b29e235e8f09010e77608cdd/aixweather/transformation_to_core_data/DWD.py#L57

So changing the source to 10-min wind data will fix the bug.

Jun-Jiang-92 commented 43 minutes ago

@MartinRaetz Thank you for your review. I'll close this issue.

Currently, I download the data only from the python script. So, for me, it is not necessary to update the webapp.

RWTH-EBC / AixWeather

Missing wind speed and wind direction by pulling DWD historical data #62