Show surface water raster layer

[robyngit]

Chandi & Rajitha have processed and uploaded the relevant raster images to the NCEAS datateam server. Here are the steps to find the surface water data:

• go to high_ice folder (/~/data/ice-wedge-polygon-data/version_01/high_ice/)
• You will see three sub-folders called: alaska, canada, russia
• Under each of the region sub-folders, there should be folders with fishnet id: like this xxx_xxx_iwp and xxx_xxx_water. The folders ending with water carry the surface water rasters ending with scene_di_xxx_xx_watermask.tif

[robyngit]

Notes:

• we would like to include this layer in a publication along with the IWP layer
• there will be more overlap between these files than there were between the IWP files. Dates can be parsed from the filename and we should prefer data from the newest files when two or more overlap.
• Data is a binary raster (water: yes/no). However, there may also be a shapefile version of this data so that we can use the same pipeline as we've already produced for the IWP dataset (TBD)

[julietcohen]

Since rearranging the Datateam server, the data is now located at: /var/data/submission/pdg/ice-wedge-polygon-data/version_01/high_ice/

The dates for the subdirectories directories are:

• alaska: March 17 2022
• canada: June 9 2022
• russia: January 21 2023

There is also another data directory that contains surface water rasters: /var/data/submission/pdg/surface_water/water_processed/ This directory's date is October 2022, so it is newer than the other directory, and it does not contain subfolders for the Alaska, Canada, and Russia. It contains 155 raster files. It would be interesting to determine if these are the same as in the other directory, and just a newer version.

[julietcohen]

Importantly, the newest version of the IWP data (more recent than version_01) is stored in /var/data/submission/pdg/ice-wedge-polygon-data/version_2023-01-31/ and within that dir we made a subdir to archive all of the files Elias used to pre-process the IWP data (after detection, before passing on to the viz team). This subdir is: pre_processing_materials/cleaning_data and contains shapefiles for surface water:

• arctic_water.shp
• lakes.shp

There is a README file as well, but it is unfortunately not helpful, because it directs the user to https://nsidc.org/data/ggd318.html which does not exist anymore. So understanding metadata for those files might need to come from Elias. Alternatively, there may be a way to determine the new URL for the page that doesn't exist anymore from the National Snow and Ice Data Center.

My best guess is that these two shapefiles should be the input to the viz workflow to produce the surface water layer, because it is the most recent version 🎉

[julietcohen]

Chandi confirmed that the water masks in raster format that came with the older version of the IWP data are not the same as the few vector waster masks that came with the newer version of the IWP dataset. So Chandi suggested that his team should send over a gdal script that converts the rasters to vectors, and that way I can do the conversion for the water mask rasters, and process those output files in the viz workflow that is configured to only ingest vectors at the moment. The team also decided it would be potentially beneficial to integrate that gdal code into the viz-staging package, if we want to pursue to route of converting rasters to vectors for other datasets as well, as an alternative to adding the ability to process rasters as input directly to the workflow.

I do want to investigate the accuracy of converting rasters to vectors, because from my experience vectorizing rasters, the output vector data is not as accurate as the data would be if it was originally in vector format. For example, adjacent polygons in a raster may be grouped into one larger polygon when it is vectorized. Which would result in less accurate statistics for # of polygons in a tile, but would be similar accuracy for % coverage of a tile.