Add custom colormap for nlcd landcover collection

anayeaye commented 1 month ago

What

Some categorical data require complex colormap definitions that exceed the allowed URL length in MCP. We can create a custom colormap for this collection, see the instructions here.

renders

"renders": {
  "dashboard": {
    "assets": [
      "landcover"
    ],
    "bidx": [
            1
        ],
        "colormap": {
          "0": "#00BFFF",
          "11": "#486DA2",
          "12": "#E7EFFC",
          "21": "#E1CDCE",
          "22": "#DC9881",
          "23": "#F10100",
          "24": "#AB0101",
          "31": "#B3AFA4",
          "41": "#6BA966",
          "42": "#1D6533",
          "43": "#BDCC93",
          "51": "#B29C46",
          "52": "#D1BB82",
          "71": "#EDECCD",
          "72": "#D0D181",
          "73": "#A4CC51",
          "74": "#82BA9D",
          "81": "#DDD83E",
          "82": "#AE7229",
          "90": "#BBD7ED",
          "95": "#71A4C1"
        },
        "nodata": 0,
    "title": "VEDA Dashboard Render Parameters",
    "resampling": "nearest"
  }
}

Working preview in environment without WAF https://staging.openveda.cloud/api/raster/collections/nlcd-annual-conus-v2/items/nlcd_2001_cog_v2/preview.png?bidx=1&assets=landcover&nodata=0&colormap=%7B%220%22%3A+%22%2300BFFF%22%2C+%2211%22%3A+%22%23486DA2%22%2C+%2212%22%3A+%22%23E7EFFC%22%2C+%2221%22%3A+%22%23E1CDCE%22%2C+%2222%22%3A+%22%23DC9881%22%2C+%2223%22%3A+%22%23F10100%22%2C+%2224%22%3A+%22%23AB0101%22%2C+%2231%22%3A+%22%23B3AFA4%22%2C+%2241%22%3A+%22%236BA966%22%2C+%2242%22%3A+%22%231D6533%22%2C+%2243%22%3A+%22%23BDCC93%22%2C+%2251%22%3A+%22%23B29C46%22%2C+%2252%22%3A+%22%23D1BB82%22%2C+%2271%22%3A+%22%23EDECCD%22%2C+%2272%22%3A+%22%23D0D181%22%2C+%2273%22%3A+%22%23A4CC51%22%2C+%2274%22%3A+%22%2382BA9D%22%2C+%2281%22%3A+%22%23DDD83E%22%2C+%2282%22%3A+%22%23AE7229%22%2C+%2290%22%3A+%22%23BBD7ED%22%2C+%2295%22%3A+%22%2371A4C1%22%7D&resampling=nearest

AC

[ ] Confirm that the URL is blocking the WAF
[ ] Custom landcover colormap supported by titiler and new url accessible behind MCP WAF

anayeaye commented 1 week ago

@vincentsarago can you help me with this custom colormap for the NLCD data? Here is what I have tried and why I know I got it wrong:

NLCD colormap

refs:

import numpy as np
from matplotlib import colors

# https://www.mrlc.gov/data/legends/national-land-cover-database-class-legend-and-description
# Category names provided here for comment only, not necessary for mapping values
nlcd_categories = {
    "11": "Open Water",
    "12": "Perennial Ice/Snow",
    "21": "Developed, Open Space",
    "22": "Developed, Low Intensity",
    "23": "Developed, Medium Intensity",
    "24": "Developed, High Intensity",
    "31": "Barren Land (Rock/Sand/Clay)",
    "41": "Deciduous Forest",
    "42": "Evergreen Forest",
    "43": "Mixed Forest",
    "51": "Dwarf Scrub",
    "52": "Shrub/Scrub",
    "71": "Grassland/Herbaceous",
    "72": "Sedge/Herbaceous",
    "73": "Lichens",
    "74": "Moss",
    "81": "Pasture/Hay",
    "82": "Cultivated Crops",
    "90": "Woody Wetlands",
    "95": "Emergent Herbaceous Wetlands"
}

nlcd_colors = {
    "11": "#486DA2",
    "12": "#E7EFFC",
    "21": "#E1CDCE",
    "22": "#DC9881",
    "23": "#F10100",
    "24": "#AB0101",
    "31": "#B3AFA4",
    "41": "#6BA966",
    "42": "#1D6533",
    "43": "#BDCC93",
    "51": "#B29C46",
    "52": "#D1BB82",
    "71": "#EDECCD",
    "72": "#D0D181",
    "73": "#A4CC51",
    "74": "#82BA9D",
    "81": "#DDD83E",
    "82": "#AE7229",
    "90": "#BBD7ED",
    "95": "#71A4C1"
}

# Create a colormap
nlcd_cmap = colors.ListedColormap(
    name = "nlcd",
    colors=[nlcd_colors[c] for c in nlcd_categories.keys()]
)

x = np.linspace(0, 1, 256)
nlcd_cmap_vals = nlcd_cmap(x)[:, :]
nlcd_cmap_uint8 = (nlcd_cmap_vals * 255).astype('uint8')
nlcd_cmap_uint8
np.save("nlcd.npy", nlcd_cmap_uint8)

This is deployed to dev.openveda.cloud and you can see the results here: https://dev.openveda.cloud/api/raster/collections/nlcd-annual-conus-v2/items/nlcd_2001_cog_v2/preview.png?bidx=1&assets=landcover&resampling=nearest&colormap_name=nlcd

There is an internal colormap in the NLCD files, however, and I can see that I got the mapping above wrong: https://openveda.cloud/api/raster/cog/viewer?url=s3://veda-data-store-staging/NLCD_V2/nlcd_2004_cog_v2.tif

I also tried starting from the colormap in the file but didn't get very far. The internal colormap looks like a good start but I haven't managed to save a usable colormap for the titiler

import rasterio
from matplotlib import colors
nlcd_filename = "/vsis3/veda-data-store-staging/NLCD_V2/nlcd_2001_cog_v2.tif"

with rasterio.open(nlcd_filename) as r:
    internal_colormap = r.colormap(1)

vincentsarago commented 1 week ago

@anayeaye you don't need a linear colormap but what I called discrete, so basically you don't need to do

# Create a colormap
nlcd_cmap = colors.ListedColormap(
    name = "nlcd",
    colors=[nlcd_colors[c] for c in nlcd_categories.keys()]
)

x = np.linspace(0, 1, 256)
nlcd_cmap_vals = nlcd_cmap(x)[:, :]
nlcd_cmap_uint8 = (nlcd_cmap_vals * 255).astype('uint8')
nlcd_cmap_uint8
np.save("nlcd.npy", nlcd_cmap_uint8)

can you register the colormap at runtime? directly by doing

from rio_tiler.colormap import cmap

nlcd_colors = {
    11: "#486DA2",
    12: "#E7EFFC",
    21: "#E1CDCE",
    22: "#DC9881",
    23: "#F10100",
    24: "#AB0101",
    31: "#B3AFA4",
    41: "#6BA966",
    42: "#1D6533",
    43: "#BDCC93",
    51: "#B29C46",
    52: "#D1BB82",
    71: "#EDECCD",
    72: "#D0D181",
    73: "#A4CC51",
    74: "#82BA9D",
    81: "#DDD83E",
    82: "#AE7229",
    90: "#BBD7ED",
    95: "#71A4C1"
}

cmap = cmap.register(
    {
        "nlcd": nlcd_colors
    }
)

if you need to go through the .npy file, there is a bug (or a feature) which will complain about your colormap not having 256 values 😓

I'll change that https://github.com/cogeotiff/rio-tiler/blob/0a8456122d3b4a256cdb63da466f32a3587df512/rio_tiler/colormap.py#L295-L299

vincentsarago commented 1 week ago

in fact I can't change the fact that npy files should have 256 values :-(

but maybe I can add .json support

ref: https://github.com/cogeotiff/rio-tiler/issues/737

anayeaye commented 1 week ago

@vincentsarago the internal colormap has 256 values (a lot of (0, 0, 0, 255)). Could we make that work for this case?

vincentsarago commented 1 week ago

@anayeaye yes, just make sure that your values are in the correct position in the numpy array

import numpy
from rio_tiler.colormap import parse_color

nlcd_colors = {
    11: "#486DA2",
    12: "#E7EFFC",
    21: "#E1CDCE",
    22: "#DC9881",
    23: "#F10100",
    24: "#AB0101",
    31: "#B3AFA4",
    41: "#6BA966",
    42: "#1D6533",
    43: "#BDCC93",
    51: "#B29C46",
    52: "#D1BB82",
    71: "#EDECCD",
    72: "#D0D181",
    73: "#A4CC51",
    74: "#82BA9D",
    81: "#DDD83E",
    82: "#AE7229",
    90: "#BBD7ED",
    95: "#71A4C1"
}

cmap = numpy.zeros((256, 4), dtype=numpy.uint8)
cmap[:] = numpy.array([0, 0, 0, 255])
for c, v in nlcd_colors.items():
    cmap[c] = numpy.array(parse_color(v))

numpy.save("nlcd.npy", cmap)

anayeaye commented 1 week ago

Thanks @vincentsarago! This worked, PR incoming

Note this example is using a _v2 collection that is only in the dev catalog for the purpose of testing the custom colormap in the dev backend https://dev.openveda.cloud/api/raster/collections/nlcd-annual-conus-v2/items/nlcd_2001_cog_v2/preview.png?bidx=1&assets=landcover&resampling=nearest&colormap_name=nlcd

NASA-IMPACT / veda-backend

Add custom colormap for nlcd landcover collection #429

What

AC

NLCD colormap