search

cropmapteam / Scotland-crop-map

This is the repository for the Scottish Government collaboration with EDINA and JNCC to produce a crop map for Scotland by developing machine learning algorithms applied to Sentinel satellite data
3 stars 0 forks source link

Check / Edit digitised RFI polygon shapefiles #34

Closed geojamesc closed 5 years ago

geojamesc commented 5 years ago

The script #31 which does the masking out of manually digitised RFI locations provided as shapefiles does some validation before doing the masking in rasterio and detected some problems with some of the shapefiles / some of the records in some of the shapefiles.

There were 51 shapefiles provided. 43 images have been processed to set to nodata, all pixels that fell within the polygons held in each shapefile

why the difference of 8?

This seems to be due to the following 3 issues with the digitised RFI polygon shapefiles which need to be looked-at; confirmed and fixed.

Issue 1: there are multiple (2) digitised RFI polygons shapefiles associated with a single image.

3 tiff files seemed to each have 2 shapefiles associated with them. In some cases duplicate RFI features have been captured, one in each shapefile as shown here:

RFI_issue1_example

The script assumes 1 shapefile per image, so processing was skipped.

So for the image S1B_20180120_132_asc_174936_175001_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.tif, there are these 2 shapefiles:

rfi/RFI_Interference_Shapefiles_Chrissy/S1B_20180120_132_asc_174936_175001_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp rfi/RFI_Interference_Shapefiles_Richard/S1B_20180120_132_asc_174936_175001_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp

So for the image S1A_20180308_30_asc_175835_175900_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.tif, there are these 2 shapefiles:

rfi/RFI_Interference_Shapefiles_Zara/ S1A_20180308_30_asc_175835_175900_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp rfi/RFI_Interference_Shapefiles_Chrissy/S1A_20180308_30_asc_175835_175900_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp

So for the image S1A_20180107_30_asc_175836_175901_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.tif, there are these 2 shapefiles:

rfi/RFI_Interference_Shapefiles_Chrissy/S1A_20180107_30_asc_175836_175901_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp rfi/RFI_Interference_Shapefiles_Richard/S1A_20180107_30_asc_175836_175901_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp

Solution - in each case the shapefiles need to be checked and where duplicate RFI regions captured, a decision be made on which is best and the features from each of the 2 shapefiles then merged into a single shapefile.

Issue 2: Some of the digitised RFI polygons shapefiles contain no records at all or all the records contain invalid geometries.

For these images, either the associated shapefile contained no records at all or the records it did contain all had invalid geometries. Processing was skipped.

So for image S1B_20180309_132_asc_174935_175000_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.tif the shapefile rfi/RFI_Interference_Shapefiles_Zara/S1B_20180309_132_asc_174935_175000_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp seems to have no records

So for image S1B_20180618_30_asc_175812_175837_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.tif the shapefile rfi/RFI_Interference_Shapefiles_Chrissy/S1B_20180618_30_asc_175812_175837_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp seems to have no records with valid geometries

Solution - can the shapefiles / digitised polygons within them be checked?

Issue3: Some of the digitised RFI polygons shapefiles contain features with invalid geometries.

These seem to mostly be self-intersections like this:

self-intersection

In such cases, the pixels associated with polygons with valid geometries were masked out as normal and a new image created. The pixels associated with polygons with invalid geometries were skipped and the pixels retained in the new image.

This can be seen in this screenshot:

RFI_issue3_example

The pixels inside the valid polygons have been masked out. The pixels inside the invalid polygon have been retained. The following shapefiles were detected as containing features with invalid geometries:

Found invalid polygon with id 8 in rfi/RFI_Interference_Shapefiles_Chrissy/S1B_20180113_30_asc_175813_175838_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp

Found invalid polygon with id 1 in rfi/RFI_Interference_Shapefiles_Zara/S1A_20180303_132_asc_175017_175042_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp

Found invalid polygon with id 10 in rfi/RFI_Interference_Shapefiles_Chrissy/S1A_20180119_30_asc_175836_175901_DV_Gamma-0_GB_OSGB_RCTK_SpkRL.shp

Solution - can the digitised polygons be checked in these shapefiles. Once done, the image can be re-processed to mask out all the RFI regions.

spatial-scales commented 5 years ago

r.e. issue 1, that's just some accidental duplication of work on our part, for consistency maybe just choose Chrissy's ones.

r.e. issues 2 and 3, Chrissy and I have checked these and hopefully fixed the geometry errors. The blank shapefile in issue 2 can just be ignored as there are no artefacts requiring removal in its associated image. I'll pop over with the corrected files for issue 3 when you're next back at your desk, and I believe Chrissy has emailed you a corrected file for issue 2.

geojamesc commented 5 years ago

ok

geojamesc commented 5 years ago

Fixed 1 more invalid geometry in one of the shapefiles and one of the shapefiles was in EPSG:4326 rather than EPSG:27700 so rasterio complained about a mismatch in SRS`s. I corrected the invalid geometry and reprojected the shapefile, then re-ran the script on this set of corrected shapefiles.