search

linz / emergency-management-tools

Source code to support emergency events
MIT License
4 stars 1 forks source link

Extract flood extents from Synthetic Aperture Radar (SAR) imaging #148

Open l0b0 opened 1 year ago

l0b0 commented 1 year ago

So Incident Management Teams at LINZ can do flood mapping, faster and more consistently, it would be useful to have a jupyter notebook for processing flood extents from common sources of SAR imagery.

Users

Inputs

Data sources

Current

Free ESA's Sentinel-1 satellite zipped images from ASF Data Search.

Non-free COGs, Sentinel Hub.

@mfwightman to review appropriate data sources.

Future

NiSAR.

Interface

JupyterLab notebook. Should not be part of sentinel2_water_extraction.ipynb, because it's dealing with very different data and algorithms.

Process

For an analysis ready image first the raw image needs have an orbit file applied, calibration applied, speckle-filtered and terrain corrected. I have documented the process in the Radar Flood Map Processing - change detection method confluence page. But of course this doesn't really cover what each of these steps is really doing. I think the best option would be if there was some python library that could do this.

I know ESA's SNAP software has a python library that has modules that can do some of the radar processing functions but I haven't used it, mainly because it seemed pretty annoying to install (i.e. I couldn't just install it via pip).

Once an image is analysis ready there are two ways to get flood extents, either using change detection or thresholding technique.

Thresholding is the easiest. Basically you assume that all values in the image below a certain value is water. The active signal that the radar sends out reflects off smooth water and not back to the sensor, so has a low return value in the image. The problem is that radar sends a signal off-nadir (off to the side at an angle) so that when there are hills you get hill-shadow where there is no return. This has the same values as water. So there is a bit of confusion around that.

Change detection is becoming my preferred option. You take two images, a before image and an image during the event, and co-register them (or create-stack or whatever the tools you are using call merging the before and during image together). Then overlay or do some band math to see where there are changes in the values returned. If there is a change to a low value from what was a not-low value then you can say this is probably water.

And generally I think that with whatever option you go with there does need to be a human visual check over the results before sending the product out to any customers.

Work out enough detail to be able to automate it.

Outputs

mfwightman commented 1 year ago

I'll start with a slight digression. Radar is short for Synthetic Aperture Radar or SAR imaging. I have chosen to use the term radar to avoid confusion between Search and Rescue which also uses the acronym SAR. This is just a heads up in case you do some research or googling.

The radar imagery we use comes from ESA's Sentinel-1 satellite. Currently it is the only open data radar available. I am not sure whether or not there is any freely available cloud optimised geotiffs, last time I checked there wasn't, but I could be wrong. I usually just get zipped images from ASF Data Search. In the future there will NiSAR but its launch keeps getting back, I'm not holding my breath for the 2024 launch. I don't think MetService or Police will be helpful.

For an analysis ready image first the raw image needs have an orbit file applied, calibration applied, speckle-filtered and terrain corrected. I have documented the process in the Radar Flood Map Processing - change detection method confluence page. But of course this doesn't really cover what each of these steps is really doing. I think the best option would be if there was some python library that could do this.

I know ESA's SNAP software has a python library that has modules that can do some of the radar processing functions but I haven't used it, mainly because it seemed pretty annoying to install (i.e. I couldn't just install it via pip).

Once an image is analysis ready there are two ways to get flood extents, either using change detection or thresholding technique.

Thresholding is the easiest. Basically you assume that all values in the image below a certain value is water. The active signal that the radar sends out reflects off smooth water and not back to the sensor, so has a low return value in the image. The problem is that radar sends a signal off-nadir (off to the side at an angle) so that when there are hills you get hill-shadow where there is no return. This has the same values as water. So there is a bit of confusion around that.

Change detection is becoming my preferred option. You take two images, a before image and an image during the event, and co-register them (or create-stack or whatever the tools you are using call merging the before and during image together). Then overlay or do some band math to see where there are changes in the values returned. If there is a change to a low value from what was a not-low value then you can say this is probably water.

And generally I think that with whatever option you go with there does need to be a human visual check over the results before sending the product out to any customers.

mfwightman commented 1 year ago

There is this requester pays COGS available here ~And this seems to be analysis ready imagery here~ This is only images over the continental US

billgeo commented 1 year ago

You can add Nic Donnelly as a user

mfwightman commented 1 year ago

Investigating two options for flood imagery, neither seems to be smooth and easy to get running

mfwightman commented 1 year ago

OpenSarToolkit seems to be failing when trying to download concurrent files, even when I only request one. Annoying.

l0b0 commented 1 year ago

FloodPy seems like a good option, but the installation instructions are bad (Using conda to install packages to python but then at the top of each script calls python3.

I don't understand why this is a bad thing - should it be calling Conda in the shebang line? That said, there are some other smells:

l0b0 commented 1 year ago

OpenSarToolKit should be pretty good, I got stuck at downloading the images and then couldn't progress forward. Its produced by ESA so should be high quality in its output. Not as fully directed toward mapping flooding as FLoodPy more just a good radar toolkit.

Yeah, there's a lot more good stuff in that repo. That said, based on a very quick review:

mfwightman commented 1 year ago

FloodPy looks like it produces a really good product, but it is very resource heavy, specifically at the Floodwater Classification step. I keep running out of memory at this step, I'm tempted to create a docker image and try on an EC2 instance. I am quite interested in seeing the product it puts out. And again I'm not sure we could package this software up into a nice thing for people to use..

billgeo commented 1 year ago

I'm wondering if we should be using something like Argo for this, or some other cloud big data processing thing, so we don't have to process this stuff locally?

l0b0 commented 1 year ago

I'm wondering if we should be using something like Argo for this, or some other cloud big data processing thing, so we don't have to process this stuff locally?

It would require a lot more development work and infrastructure (nothing right now), and adds a big barrier to entry for anyone not familiar with cloud services. I'd suggest focusing on making it work locally first, and only move it to the cloud once we're sure we can't achieve decent performance locally.