Incorporate radar imagery

[astrobokonon]

A plausible path forward exists because NEXRAD data are also in an AWS open data bucket (https://registry.opendata.aws/noaa-nexrad/) but I haven't looked at it ever so I don't know how transferrable something like GOESMcGOESface really is.

Also, I don't know the relative priority of this vs. just getting something in place for initial use. I suspect it's pretty high, but I'm leaving here for the current (alpha) iteration.

[astrobokonon]

I'm officially bumping this up, based on what I saw today while out at the site.

On one screen, it was: allsky camera image, a browser with a choice of tabs showing GOES-16 LWIR, old GOES IR, and the NWS radar

On the other screen, it was: WRS, ICM, MTS, and MTM.

Based on those above, I think I need to get the NEXRAD source explored as soon as I'm done standing up and integrating the Bokeh plots.

[astrobokonon]

Open question:

Can the radar imagery be shown as an (optional?) overlay in the existing weather window, or should it be a distinct entity?

Potential hurdles:

• Differing time frames for NEXRAD vs. GOES-16 data. It's not clear to me that this is easily solved?
• Increase in visual clutter might be a killer

[astrobokonon]

I've settled on a separate HAniS window, for now, because the time base problem seems like too much of a problem in the way I've set things up. It's just not happening right now, and if someone wants/demands it they'll have to be the one to figure that out.

The AWS bucket is really called 'noaa-nexrad-level2' and is organized by year/month/date/stationID so it's pretty similar to the GOES ones. The station we want is 'KFSX' since that's Flagstaff weather station's handle.

Filenames are simple: KFSX20190515_000056_V06 so it's stationYYYYMMDD_HHMMSS_V06. Sometimes there's a file with a suffix _MDM which I'm going to filter out for now because I don't know what that is. The AWS description says that the files are gzipped, but my S3 browser doesn't show that; not sure if the extension is missing or if the files just aren't compressed anymore for whatever reason.

The seem to range somewhere between 3-8 MB, probably depending on how much was picked up on the radar.

[astrobokonon]

Note: radar data is pretty complex. There's a lot that can go awry, especially if I apply any sort of filtering. Because of that, I'm going to fall back and only display the raw data, which will contain radar shadows and noise from ground clutter, bugs, and birds.

Basically, I want to put a big notice on the thing that says users should use their eyeballs and other contextual information to verify what's around.

[astrobokonon]

Note for posterity: the NEXRAD data are "Level 2" data files, as described here: https://www.roc.noaa.gov/wsr88d/Level_II/Level2Info.aspx

This means that "composite" reflectivity is unavailable, since that's a "Level 3" data product. I can't find Level 3 data anywhere on AWS, so that's fine for now and I'll just plot the base reflectivity.

[astrobokonon]

Initial attempt - no filtering applied and no map information, but it matches up with what I see on NOAA's site. Worth noting that NOAA does apply some sort of filtering on their maps.

[astrobokonon]

If I want to investigate fancy filtering, https://github.com/jzbontar/steiner_smith is worth another look. It downsamples to the given grid size, and I found that it was filtering out some stuff that I felt looked real, but it's probably worth fiddling with more. For now, I am actually going to apply the simple de-bugging I've found referenced in a few other places.

[astrobokonon]

Mostly complete result. I settled on the filtering as shown in this notebook: nexrad_sample.ipynb because it's simple and we're not doing actual meterology. I did lower the initial threshold so it's not as harsh at the expense of some more clutter in the map.

I'm really just fiddling with the colormap before I call this good to put into testing over the weekend.

[astrobokonon]

Reminder for the winter; I'm explicitly filtering out values < 10 dBZ, so I'll likely miss light snow and stuff like that. Some details to read here: https://avwxworkshops.com/etips/01-06-10.html.

[astrobokonon]

Ok, got the NWS table in place and here's how things look.

Unfiltered:

Filtered (dBZ > 0):

[astrobokonon]

"Final" filtering values for now; the maps will exclude volumes that match any of the following:

Reflectivity < 5 dBZ IF the radar scan mode is NOT one of the "clear air" types (VCP == 31 || 32 || 35) Differential Reflectivity < -2.0 or > 2.0 dBZ Cross Correlation Ratio < 0.90

[astrobokonon]

Left it running after I left work tonight, and it seems to be going pretty well. It does indeed transition to minimal filtering once VCP == 35, which makes sense for NWS to transition to faint snow/precip. searching when the bulk of the storm passes.

[astrobokonon]

Camelot@690141a put into testing for a longer run, I have a suspicion there will be more bugs to hunt but it'll need some more runtime to see.

DockerDev@e2342ab AND DockerDev@35806dd AND DockerDev@6a62959 contain the updates to the dockEnvFile.sh and docker-compose file. Remember there is another config file that is not committed that needed updates too, nginx-app.conf

[astrobokonon]

Took some fiddling and fighting with HAniS, but it's incorporated into the main alpha as of 970faccd215eaaab7b061eb4b7c054b90b21cc84.