Weather radar

[jdemaeyer]

With v2.0.2, a new /radar endpoint landed in Bright Sky. :tada:

This endpoint contains

• rainfall radar data
• on a 1200 km (height) x 1100 km (width) grid
• with each pixel corresponding to 1 km²
• in 5-minute intervals
• including a forecast for the next two hours.

This data comes from the DWD's RV radar composite product.

Please feel very welcome to add feedback and suggestions to this issue!

Quickstart

This request will get you radar data near Münster, reaching 200 km to the East/West/North/South, as a two-dimensional grid of integers:

https://api.brightsky.dev/radar?lat=52&lon=7.6&format=plain

API structure

• The new endpoint is available at https://api.brightsky.dev/radar

• The response looks roughly like:

  {
    "radar": [
      {
        "timestamp": "2023-05-09T11:05:00+00:00",
        "source": "RADOLAN::RV::2023-05-09T11:05:00+00:00",
        "precipitation_5": "... base64-encoded bytestring or array of integers, see below ..."
      },
      ...
    ],
    "geometry": {
      "type": "Geometry",
      "coordinates": [
        [1.4633,55.86209],
        [3.56699,45.69643],
        [16.58087,45.68461],
        [18.73162,55.84544],
      ]
    }
  }

• Allowed parameters (all of them optional):

  • format: determines how the precipitation data is encoded into the precipitation_5 field (default: compressed):
  • compressed: base64-encoded, zlib-compressed bytestring of 2-byte integers
  • bytes: base64-encoded bytestring of 2-byte integers
  • plain: Nested array of integers
  • bbox: bounding box top, left, bottom, right in pixels, edges are inclusive (default: full 1200x1100 grid)
  • lat / lon / distance: alternative way to set a bounding box, where lat / lon will lay in the center pixel of the returned radar data, which will reach distance meters to each side of this pixel (see example request below). distance can be omitted and defaults to 200000 (i.e. 200 km). The exact x-y-position of the supplied lat/lon will be returned as latlon_position
  • date: ISO8601 timestamp of first record (default: 1 hour before latest measurement)
  • last_date: ISO8601 timestamp of last record (default: 2 hours after date)
  • tz: timezone to be used for timestamps (default: UTC)

Parsing examples

The radar data is quite big (naively unpacking the default 25-frames response into Python integer arrays will eat roughly 125 MB of memory), so use bbox whenever you can.

compressed format

With Python using numpy:

import base64
import zlib

import numpy as np
import requests

resp = requests.get('https://api.brightsky.dev/radar')
raw = resp.json()['radar'][0]['precipitation_5']
raw_bytes = zlib.decompress(base64.b64decode(raw))

data = np.frombuffer(
    raw_bytes,
    dtype='i2',
).reshape(
    # Adjust this to the height/width of your bbox
    (1200, 1100),
)

With Python using the standard library's array:

import array

# [... load raw_bytes as above ...]

data = array.array('H')
data.frombytes(raw_bytes)
data = [
  # Adjust `1200` and `1100` to the height/width of your bbox
  data[row*1100:(row+1)*1100]
  for row in range(1200)
]

Simple plot using matplotlib:

import matplotlib.pyplot as plt

# [... load data as above ...]

plt.imshow(data, vmax=50)
plt.show()

bytes format

Same as for compressed, but add ?format=bytes to the URL and remove the call to zlib.decompress, using just raw_bytes = base64.b64decode(raw) instead.

plain format

This is obviously a lot simpler than the compressed format. It is, however, also a lot slower. Nonetheless, if you have a small-ish bbox the performance difference becomes manageable, so just using the plain format and not having to deal with unpacking logic can be a good option in this case.

With Python:

import requests

resp = requests.get('https://api.brightsky.dev/radar?format=plain')
data = resp.json()['radar'][0]['precipitation_5']

Content

• The grid is a polar stereographic projection of Germany and the regions bordering it. This is different from the mercator projection used for most consumer-facing maps like OpenStreetMap or Google Maps, and overlaying the radar data onto such a map without conversion will be inaccurate!
• The DWD data does not cover the whole grid! Many areas near the edges will always be 0.
• Values represent 0.01 mm / 5 min. I.e., if a pixel has a value of 45, then 0.45 mm of precipitation fell in the corresponding square kilometer in the past five minutes.
• The four corners of the grid are as follows:
  • Northwest: Latitude 55.86208711, Longitude 1.463301510
  • Northeast: Latitude 55.84543856, Longitude 18.73161645
  • Southeast: Latitude 45.68460578, Longitude 16.58086935
  • Southwest: Latitude 45.69642538, Longitude 3.566994635
• The latitude/longitude for the center of any grid cell (pixel) can be found in this file.

You can find details and more information in the DWD's RV product info (German only). Below is an example visualization of the rainfall radar data taken from this document, using the correct projection and showing the radar coverage:

Notes and Questions for Feedback

• This is an experimental feature and the interface is very much subject to change. Please regularly check this issue if you're using it.
• Use a bbox and the default compressed format if possible -- this'll get you the fastest response times by far and reduce load on the server. If you have a small-ish bounding box (e.g. 250 x 250 pixels), using the plain format should be fine.
• I'm not sure how useful the bytes format is. The idea is to save the hassle of parsing a giant JSON array on the client side, but maybe that's actually not too bad?
• The polar-stereographic projection that this data uses makes a lot of sense from a data perspective (as each pixel represents the same area) but is rarely used in maps. If you're looking for tiles you can overlay on a map, take a look at the dwd:RV-Produkt layer on the DWD's open GeoServer.
• The conversion between pixel x/y coordinates and lat/lon coordinates is quite unwieldy. Many users will probably just want to know "what's the rainfall like close to my location", so maybe alternative parameters for setting the bounding box by supplying the lat/lon of its center and a maximum distance could be a simple interface for this?

Resources

• Coordinates for the center of every grid cell
• Script that generated the above coordinates
• Raw data on the Open Data Server
• Details on the RV product (German)
• Visualization of current rainfall radar
• General info on DWD radar products (German)
• Radar status (German)
• DWD notifications for radar products (German)

[poetaster]

As you suggest in your last comment on conversion, I'm thinking of a first simple bounding box from the current location (or a user supplied location). I'm rendering in QT and suppose I could render directly to canvas (the QML type derived from javascript). If I understand correctly, that would be a trivial task of just taking the matrix for the area (say 512x512 pixels) and directly rendering the floats as 'my favourite color pallette'?

[ntadej]

Note that DWD also has a GeoServer instance which is much nicer to be used (and if I understand correctly can be used directly by clients). Then supported mapping engines can render it more efficiently (although it's still bitmaps). I use that in my Qt-based app.

[poetaster]

Then supported mapping engines can render it more efficiently (although it's still bitmaps).

I'm currently using a webview using leaflet and the data from rainviewer.com (which consumes the DWD data) but it's both slow and, obviously, not native. I guess I could go to rendering something like the Composite openlayer view: https://maps.dwd.de/geoserver/dwd/wms?service=WMS&version=1.1.0&request=GetMap&layers=dwd%3AWN-Produkt&bbox=-543.462%2C-4808.645%2C556.538%2C-3608.645&width=703&height=768&srs=EPSG%3A1000001&styles=&format=application/openlayers#toggle

Is that what you mean? I don't actually want to use a mapping engine since I'm a mobile user and want to conserve resources. For directly rendering, say, via leaflet.js I think you're obviously correct, but what if I just want a simple/naive representation?

[jdemaeyer]

If I understand correctly, that would be a trivial task of just taking the matrix for the area (say 512x512 pixels) and directly rendering the floats as 'my favourite color pallette'?

That's how I would do it, yeah. I am however not very experienced in handling map data, unfortunately. My intuition is this: If the canvas already contains a map that you render on top of, and you aligned the center of the map (i.e. the user location) to the center of the radar data matrix, the location mismatch should be negligible for the center pixels but will become increasingly bigger for locations away from the center—I don't know by how much, though. However, for a naive "I just wanna know if there's rain coming up at my location" that's probably Good Enough™?

[jdemaeyer]

Before it gets lost in the wall of text I posted above: Here is a file containing the center coordinates for each pixel.

I will probably use those (or rather, the mathematical conversion outlined in this document on page 13) to implement center_lat/center_lon and width/height parameters if those requests turn out to be a major use case.

[poetaster]

However, for a naive "I just wanna know if there's rain coming up at my location" that's probably Good Enough™?

I think I'll need to compare the actual data. It could well be that overlay data from DWD is smaller, but the additional map js/tile data again competes. I'll have to try it to see how it looks. Thanks!

[jdemaeyer]

Sounds great, let me know how it goes! :)

[jdemaeyer]

I've added lat, lon, and distance parameters as an easier way to set a bounding box. This request will return a pixel grid with Münster in its center, reaching 200 km in every direction:

https://api.brightsky.dev/radar?lat=52&lon=7.6&format=plain

(The distance in each direction can be adjusted by supplying distance in meters, the default is 200000)

[AndiLeni]

I really like to see this feature. However, I can not see how I may use this to easily show a map to the user.

What I really would like to see is this Radardata as a geojson or SVG image, since the official dwd map proxy only provides them as png/jpg and not in a scalable format.

But thanks nonetheless for this awesome feature!

[jdemaeyer]

Thanks for the feedback @AndiLeni. I was afraid this feature will not see much use as it's not easy to integrate into map applications in its current form (I've had a heated discussion with @ubergesundheit over lunch about this as well :wave:). Bright Sky is very much a service for numerical weather data, and I fear leaving that core competence and making it into a "numerical weather data but then also some vector map tiles service" will open the gate to a whole range of development/maintenance nightmares.

That being said, providing the radar data as GeoJSON seems close enough to what we do currently that we could sneak it in. What would radar data look like in this format, a giant list of Polygon features, with one polygon for each grid cell (pixel) of the radar data?

[AndiLeni]

@jdemaeyer I am honestly not sure what might be the best. Aggregating several polygons with the same data-information to a larger one would be the most ideal way to keep the geojson simple and lightweight. But this would probably require a lot of preprocessing (?)

It might be good to try it with leaflet or openlayers, so see how performant or good it is.

Would it make you a lot of work to deliver geojson for evaluating this?

[firewiremb]

Hello jedmayer, I'm excited about the new endpoint /radar. Thank you for your great work.

I have a question about this: Would it be possible to take the satellite-based cloud cover data into account in addition to the precipitation data?

The DWD seems to provide relevant data here:

https://www.dwd.de/DE/dienstleistungen/wolken_niederschlagsart/wolken_niederschlagsart.html https://opendata.dwd.de/weather/satellite/clouds/)

[jdemaeyer]

Hi @firewiremb. I would love to have that map of cloud coverage in Bright Sky!

Unfortunately, I wasn't able to find data for the detailed cloud coverage in Germany you linked to on the Open Data server. (Which is not to say that it doesn't exist there, only that I wasn't smart enough to find it so far ;))

The two folders under /weather/satellite/clouds are the cloud top height for Western and Central Europe and the thunderstorm severity for all of Europe and Africa. These two are not enriched by radar data and are quite coarse for Germany: ~ 300x250 pixels and 180x150 pixels, respectively. I'm unsure how exactly the CTH maps to cloud coverage (as seen from the ground).

Is that data also of interest to you?

[brgrp]

I've added lat, lon, and distance parameters as an easier way to set a bounding box. This request will return a pixel grid with Münster in its center, reaching 200 km in every direction:

https://api.brightsky.dev/radar?lat=52&lon=7.6&format=plain

(The distance in each direction can be adjusted by supplying distance in meters, the default is 200000)

Hi, I just tested the lat,lon,distance parameters and it works great! I am just wondering now, how to get the exact GPS coordinates for all the corners.

I could probably estimate it roughly using distance and projections, but the information should be much more accurately available on the server anyway. My suggestion would be to include the GPS bounding box in the response so that the user can more easily and accurately match the data to the map data. Or am I missing something?

[jdemaeyer]

Thanks for the feedback @codegrafix. The response already contains the bounding box under the geometry key :)

[brgrp]

Thanks for the fast reply! It worked.

Any idea why the map is rotated? Does this come from the projection? Cheers

[jdemaeyer]

Awesome screenshot, thanks! Also note that the lat/lon you supply with the request is very unlikely to be located at the exact center of a DWD radar grid cell, and thus the center of the bounding box is somewhere between 0–500 m off in each direction from your lat/lon. The latlon_position field contains the (sub-)pixel coordinates for your lat/lon.

Any idea why the map is rotated? Does this come from the projection?

I assume yes. Almost everything that's facing consumers is using the Mercator projection, while the DWD uses a stereographic projection for the radar data. From my understanding (but I am very much out of my depth here) this is very helpful for data analysis, as each pixel corresponds to exactly 1 km² – making calculations like "total rainfall for this larger area" straightforward. Unfortunately that comes at the price of being a headache to overlay onto a Mercator-projected map...

[brgrp]

Thanks for the hint! I played a little bit around with the data and decided to build a metric grid (UTM).

The 3D visualization shows the current rain in Munich, where the height of each gridzell shows the precipitation per 1km² grid cell.

If you are interested in the visualization, I used keplerGl and the output file of my script: file.csv.

[jdemaeyer]

Closing as this is now officially part of the API :)

[jdemaeyer]

Demo of radar visualization (including reprojection onto Mercator): live, source

Example Javascript to decompress the radar data when using the default format (using pako):

fetch(
  'https://api.brightsky.dev/radar'
).then((resp) => resp.json()
).then((respData) => {
  const raw = respData.radar[0].precipitation_5;
  const compressed = Uint8Array.from(atob(raw), c => c.charCodeAt(0));
  const rawBytes = pako.inflate(compressed).buffer;
  const precipitation = new Uint16Array(rawBytes);
});