Wind-Speed forecast data not correctly scraped

[seb2010]

I am diving into the weather data to create wind-generation power forecasts. While doing that, I am scraping the historical-data from the DWD OpenWEB-sources itself. Because the KML-data is too massive, I want to rely on BrightSky API.

Now when getting the forecast data from BrightSky, I observed a massive disconnect from the values from DWD and BrightSky for the wind speed. The BrightSky data is magnitutes higher and also shows weird reoccuring pattern, bascially not changing from day-to-day.

I then looked into the KML data (e.g. for Station 10303, DWD-Station 15813) directly, and the data is totally different from what is shown in BrightSky. This may be due to scraping delay of the BrightSky DB, but it is way off. And that is basically true for all stations. Other values, like air-pressure and wind-direction are fine. The wind-speed scraping seems faulty.

Or am I looking at the data wrong?

[seb2010]

I had not seen the m/s to km/h conversion

[jdemaeyer]

Hi @seb2010, thanks for reaching out and for the update. Feel free to re-open if you have other questions or if you think there's something wrong with the data even after taking the unit conversion into account :)

[seb2010]

Hi @jdemaeyer, thanks for replying so quickly. I am still struggeling with the units for solar. As said, I am scraping the historic data myself and getting the forecats mosmix data from Birghtsky. But the unit the data is provided does noch match and even when looking closer at the unit that should be provided, the resulting factor is not right.' My assumptions:

• CDC data: J/cm²
• Mosmix-data: kJ/m²
• Brightsky-parse factor for mosmix "kj_per_m2_to_j_per_m2": 1000 --> kJ/m² 1000 = J/m2
• Brightsky result: kJ/m² --> to convert to J/cm² additional factor of / 10.000 is needed. This results in a combined mosmix-factor of 1000/10000 = 0.1 to get J/cm² comparable to CDC data.

Either way, I do not get results that look right. If I just add a factor of *100/3.6 to the Brightsky data it looks comparable to the CDC 10minute data, but I do not understand why. Can you help explain (german is also fine)?

[jdemaeyer]

Yeah the units are a tad bit annoying, especially since the DWD is not using them consistently between their products. As you point out, the units in the source files are:

• kJ / m² for MOSMIX (the only field in the MOSMIX data that does not use SI units...),
• J / m² for SYNOP messages (which Bright Sky uses for the current_weather endpoint),
• "average W / m²" for current observations (i.e., unlike all other files they report irradiance, not irradiation),
• J / cm² for historical observations (CDC).

Bright Sky's parsing abstracts all these different units away and stores everything in SI units, i.e. J / m² for solar irradiation. However, what you get back when querying (I presume that's what you titled "Brightsky result") depends on how you set the units query parameter. If you don't supply it (or supply the default value dwd), Bright Sky will return values in units that are most common in consumer-level weather apps, in this case kWh / m².

So to convert Bright Sky's result to J / cm², as used in the CDC records, you can either:

• Continue using units=dwd and multiply the returned value by 360 (which is "multiply by 3,600,000" to get from kWh to J (=Ws), then "divide by 10,000" to get from 1/m² to 1/cm²)
• Supply units=si and divide by 10,000 (to get from 1/m² to 1/cm²)

Note that the DWD (and consequently Bright Sky) reports irradiation, i.e. total incoming energy over a given time period per area, not irradiance, i.e. momentary power per area. So when you compare to the CDC 10-minute data, you will need to add up all six data points of an hour (giving you the total incoming energy for that hour) before comparing to the hourly value returned by Bright Sky.

Example:

• The CDC file for Münster/Osnabrück contains the following irradiation data for 2023-12-11 between 11:00–12:00 UTC:

  MESS_DATUM      GS_10
  202312111100      2.9
  202312111110      3.8
  202312111120      2.4
  202312111130      2.1
  202312111140      2.1
  202312111150      3.2

• Summing up gives us a total irradiation of 16.5 J / cm² (note that there's an annoying inconsistency in the CDC files where the values correspond to the irradiation for the next ten minutes, not the previous ten minutes)
• Bright Sky returns solar=0.046 (kWh / m²) for 12:00 UTC yesterday in Münster, when using the default dwd units. Multiplying by 360 gives us 16.56 J / cm² (with a small rounding error).
• Alternatively, when querying with units=si, we get solar=165000 (J / m²). Dividing by 10,000 gives us 16.5 J / cm².

Apologies for the long reply, I had to sort this out for myself (again) as well ;) Hope it helps!