Closed jdemaeyer closed 1 year ago
As you suggest in your last comment on conversion, I'm thinking of a first simple bounding box from the current location (or a user supplied location). I'm rendering in QT and suppose I could render directly to canvas (the QML type derived from javascript). If I understand correctly, that would be a trivial task of just taking the matrix for the area (say 512x512 pixels) and directly rendering the floats as 'my favourite color pallette'?
Note that DWD also has a GeoServer instance which is much nicer to be used (and if I understand correctly can be used directly by clients). Then supported mapping engines can render it more efficiently (although it's still bitmaps). I use that in my Qt-based app.
Then supported mapping engines can render it more efficiently (although it's still bitmaps).
I'm currently using a webview using leaflet and the data from rainviewer.com (which consumes the DWD data) but it's both slow and, obviously, not native. I guess I could go to rendering something like the Composite openlayer view: https://maps.dwd.de/geoserver/dwd/wms?service=WMS&version=1.1.0&request=GetMap&layers=dwd%3AWN-Produkt&bbox=-543.462%2C-4808.645%2C556.538%2C-3608.645&width=703&height=768&srs=EPSG%3A1000001&styles=&format=application/openlayers#toggle
Is that what you mean? I don't actually want to use a mapping engine since I'm a mobile user and want to conserve resources. For directly rendering, say, via leaflet.js I think you're obviously correct, but what if I just want a simple/naive representation?
If I understand correctly, that would be a trivial task of just taking the matrix for the area (say 512x512 pixels) and directly rendering the floats as 'my favourite color pallette'?
That's how I would do it, yeah. I am however not very experienced in handling map data, unfortunately. My intuition is this: If the canvas already contains a map that you render on top of, and you aligned the center of the map (i.e. the user location) to the center of the radar data matrix, the location mismatch should be negligible for the center pixels but will become increasingly bigger for locations away from the center—I don't know by how much, though. However, for a naive "I just wanna know if there's rain coming up at my location" that's probably Good Enough™?
Before it gets lost in the wall of text I posted above: Here is a file containing the center coordinates for each pixel.
I will probably use those (or rather, the mathematical conversion outlined in this document on page 13) to implement center_lat
/center_lon
and width
/height
parameters if those requests turn out to be a major use case.
However, for a naive "I just wanna know if there's rain coming up at my location" that's probably Good Enough™?
I think I'll need to compare the actual data. It could well be that overlay data from DWD is smaller, but the additional map js/tile data again competes. I'll have to try it to see how it looks. Thanks!
Sounds great, let me know how it goes! :)
I've added lat
, lon
, and distance
parameters as an easier way to set a bounding box. This request will return a pixel grid with Münster in its center, reaching 200 km in every direction:
https://api.brightsky.dev/radar?lat=52&lon=7.6&format=plain
(The distance in each direction can be adjusted by supplying distance
in meters, the default is 200000
)
I really like to see this feature. However, I can not see how I may use this to easily show a map to the user.
What I really would like to see is this Radardata as a geojson or SVG image, since the official dwd map proxy only provides them as png/jpg and not in a scalable format.
But thanks nonetheless for this awesome feature!
Thanks for the feedback @AndiLeni. I was afraid this feature will not see much use as it's not easy to integrate into map applications in its current form (I've had a heated discussion with @ubergesundheit over lunch about this as well :wave:). Bright Sky is very much a service for numerical weather data, and I fear leaving that core competence and making it into a "numerical weather data but then also some vector map tiles service" will open the gate to a whole range of development/maintenance nightmares.
That being said, providing the radar data as GeoJSON seems close enough to what we do currently that we could sneak it in. What would radar data look like in this format, a giant list of Polygon
features, with one polygon for each grid cell (pixel) of the radar data?
@jdemaeyer I am honestly not sure what might be the best. Aggregating several polygons with the same data-information to a larger one would be the most ideal way to keep the geojson simple and lightweight. But this would probably require a lot of preprocessing (?)
It might be good to try it with leaflet or openlayers, so see how performant or good it is.
Would it make you a lot of work to deliver geojson for evaluating this?
Hello jedmayer, I'm excited about the new endpoint /radar. Thank you for your great work.
I have a question about this: Would it be possible to take the satellite-based cloud cover data into account in addition to the precipitation data?
The DWD seems to provide relevant data here:
https://www.dwd.de/DE/dienstleistungen/wolken_niederschlagsart/wolken_niederschlagsart.html https://opendata.dwd.de/weather/satellite/clouds/)
Hi @firewiremb. I would love to have that map of cloud coverage in Bright Sky!
Unfortunately, I wasn't able to find data for the detailed cloud coverage in Germany you linked to on the Open Data server. (Which is not to say that it doesn't exist there, only that I wasn't smart enough to find it so far ;))
The two folders under /weather/satellite/clouds
are the cloud top height for Western and Central Europe and the thunderstorm severity for all of Europe and Africa. These two are not enriched by radar data and are quite coarse for Germany: ~ 300x250 pixels and 180x150 pixels, respectively. I'm unsure how exactly the CTH maps to cloud coverage (as seen from the ground).
Is that data also of interest to you?
I've added
lat
,lon
, anddistance
parameters as an easier way to set a bounding box. This request will return a pixel grid with Münster in its center, reaching 200 km in every direction:
https://api.brightsky.dev/radar?lat=52&lon=7.6&format=plain
(The distance in each direction can be adjusted by supplying
distance
in meters, the default is200000
)
Hi, I just tested the lat,lon,distance parameters and it works great! I am just wondering now, how to get the exact GPS coordinates for all the corners.
I could probably estimate it roughly using distance and projections, but the information should be much more accurately available on the server anyway. My suggestion would be to include the GPS bounding box in the response so that the user can more easily and accurately match the data to the map data. Or am I missing something?
Thanks for the feedback @codegrafix. The response already contains the bounding box under the geometry
key :)
Thanks for the fast reply! It worked.
Any idea why the map is rotated? Does this come from the projection? Cheers
Awesome screenshot, thanks! Also note that the lat/lon you supply with the request is very unlikely to be located at the exact center of a DWD radar grid cell, and thus the center of the bounding box is somewhere between 0–500 m off in each direction from your lat/lon. The latlon_position
field contains the (sub-)pixel coordinates for your lat/lon.
Any idea why the map is rotated? Does this come from the projection?
I assume yes. Almost everything that's facing consumers is using the Mercator projection, while the DWD uses a stereographic projection for the radar data. From my understanding (but I am very much out of my depth here) this is very helpful for data analysis, as each pixel corresponds to exactly 1 km² – making calculations like "total rainfall for this larger area" straightforward. Unfortunately that comes at the price of being a headache to overlay onto a Mercator-projected map...
Thanks for the hint! I played a little bit around with the data and decided to build a metric grid (UTM).
The 3D visualization shows the current rain in Munich, where the height of each gridzell shows the precipitation per 1km² grid cell.
If you are interested in the visualization, I used keplerGl and the output file of my script: file.csv.
Closing as this is now officially part of the API :)
Demo of radar visualization (including reprojection onto Mercator): live, source
Example Javascript to decompress the radar data when using the default format
(using pako):
fetch(
'https://api.brightsky.dev/radar'
).then((resp) => resp.json()
).then((respData) => {
const raw = respData.radar[0].precipitation_5;
const compressed = Uint8Array.from(atob(raw), c => c.charCodeAt(0));
const rawBytes = pako.inflate(compressed).buffer;
const precipitation = new Uint16Array(rawBytes);
});
With v2.0.2, a new
/radar
endpoint landed in Bright Sky. :tada:This endpoint contains
This data comes from the DWD's RV radar composite product.
Please feel very welcome to add feedback and suggestions to this issue!
Quickstart
This request will get you radar data near Münster, reaching 200 km to the East/West/North/South, as a two-dimensional grid of integers:
https://api.brightsky.dev/radar?lat=52&lon=7.6&format=plain
API structure
The new endpoint is available at
https://api.brightsky.dev/radar
The response looks roughly like:
Allowed parameters (all of them optional):
format
: determines how the precipitation data is encoded into theprecipitation_5
field (default:compressed
):compressed
: base64-encoded, zlib-compressed bytestring of 2-byte integersbytes
: base64-encoded bytestring of 2-byte integersplain
: Nested array of integersbbox
: bounding boxtop, left, bottom, right
in pixels, edges are inclusive (default: full 1200x1100 grid)lat
/lon
/distance
: alternative way to set a bounding box, wherelat
/lon
will lay in the center pixel of the returned radar data, which will reachdistance
meters to each side of this pixel (see example request below).distance
can be omitted and defaults to200000
(i.e. 200 km). The exact x-y-position of the suppliedlat
/lon
will be returned aslatlon_position
date
: ISO8601 timestamp of first record (default: 1 hour before latest measurement)last_date
: ISO8601 timestamp of last record (default: 2 hours afterdate
)tz
: timezone to be used for timestamps (default: UTC)Parsing examples
The radar data is quite big (naively unpacking the default 25-frames response into Python integer arrays will eat roughly 125 MB of memory), so use
bbox
whenever you can.compressed
formatWith Python using
numpy
:With Python using the standard library's
array
:Simple plot using
matplotlib
:bytes
formatSame as for
compressed
, but add?format=bytes
to the URL and remove the call tozlib.decompress
, using justraw_bytes = base64.b64decode(raw)
instead.plain
formatThis is obviously a lot simpler than the
compressed
format. It is, however, also a lot slower. Nonetheless, if you have a small-ishbbox
the performance difference becomes manageable, so just using theplain
format and not having to deal with unpacking logic can be a good option in this case.With Python:
Content
0
.45
, then 0.45 mm of precipitation fell in the corresponding square kilometer in the past five minutes.55.86208711
, Longitude1.463301510
55.84543856
, Longitude18.73161645
45.68460578
, Longitude16.58086935
45.69642538
, Longitude3.566994635
You can find details and more information in the DWD's
RV product info
(German only). Below is an example visualization of the rainfall radar data taken from this document, using the correct projection and showing the radar coverage:Notes and Questions for Feedback
bbox
and the defaultcompressed
format if possible -- this'll get you the fastest response times by far and reduce load on the server. If you have a small-ish bounding box (e.g. 250 x 250 pixels), using theplain
format should be fine.bytes
format is. The idea is to save the hassle of parsing a giant JSON array on the client side, but maybe that's actually not too bad?dwd:RV-Produkt
layer on the DWD's open GeoServer.Resources
RV
product (German)