Support raster overviews

Proposed Changes

Support raster overviews.
Include per-band stats in metadata.
Include bands colorinterp field.

Pull Request Checklist

[x] I have tested the changes locally
[x] I have added tests to cover my changes (if applicable)
[ ] I have updated the documentation (if applicable)

Additional Information

We used different examples to test a variety of general cases.

Note the number of overviews can be seen in metadata from the difference between maxresolution and minresolution values, as is defined as following:

        metadata["minresolution"] = resolution - len(raster_dataset.overviews(1))
        metadata["maxresolution"] = resolution

cartocog_3byte_60cm_small.tif (1.3MB)

Trying to upload the file:

    carto bigquery upload --file_path tiffs/cartocog_3byte_60cm_small.tif --project cartodb-on-gcp-backend-team --dataset juanra --table cartocog_3byte_60cm_small --overwrite --band 1 --band 2 --band 3  --token XXX

We get an error: Error uploading to BigQuery: The input raster must be a GoogleMapsCompatible raster. So we need to pre-process the file with gdalwarp, using the recommended options:

    gdalwarp -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 \
    tiffs/cartocog_3byte_60cm_small_nodata.tif \
    tiffs/cartocog_3byte_60cm_small_gmaps.tif

Then uploading the resulting file works fine:

    carto bigquery upload --file_path tiffs/cartocog_3byte_60cm_small_gmaps.tif --project cartodb-on-gcp-backend-team --dataset juanra --table cartocog_3byte_60cm_small --overwrite --band 1 --band 2 --band 3 --band 4  --token XXX

Visualizing with deck.gl:

Using this for the layer config:

    source: rasterSource,
    tableName: 'cartodb-on-gcp-backend-team.juanra.cartocog_3byte_60cm_small',
    getFillColor: (d) => {
      const { band_1, band_2, band_3, band_4 } = d.properties;
      return [band_1, band_2, band_3, band_4 === 0 || (band_1 === 0 && band_2 === 0 && band_3) === 0 ? 0 : 255]
    },

cartocog_3byte_30cm_small_pixel.tif (2.7MB)

Process the file with gdalwarp including recommended options:

    gdalwarp -srcnodata 0 -dstnodata 0 -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 \
    tiffs/cartocog_3byte_30cm_small_pixel.tif \
    tiffs/cartocog_3byte_30cm_small_pixel_gmaps.tif

Then upload the results with raster-loader:

    carto bigquery upload --file_path tiffs/cartocog_3byte_30cm_small_pixel_gmaps.tif --project cartodb-on-gcp-backend-team --dataset juanra --table cartocog_3byte_30cm_small_pixel --overwrite --band 1 --band 2 --band 3 --BAND 4 --token XXX

Raise an error: Error: Error uploading to BigQuery: Input raster pixel resolution exceeds the max supported resolution of 26.

So remapping with -co ZOOM_LEVEL=17 option it works fine:

    gdalwarp -srcnodata 0 -dstnodata 0 -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 -co ZOOM_LEVEL=17 \
    tiffs/cartocog_3byte_30cm_small_pixel.tif \
    tiffs/cartocog_3byte_30cm_small_pixel_gmaps.tif

Using in this case:

getFillColor: (d) => {
      const { band_1, band_2, band_3, band_4 } = d.properties;
      return [band_1, band_2, band_3, band_4 === 0 || (band_1 === 0 && band_2 === 0 && band_3 === 0) ? 0 : 255]
    },

cartocog_byte_10km_MOD12Q1_Land_Cover_Type2.tif (512KB)

    gdalwarp -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 \
    tiffs/cartocog_byte_10km_MOD12Q1_Land_Cover_Type2.tif \
    tiffs/cartocog_byte_10km_MOD12Q1_Land_Cover_Type2_gmaps.tif

    carto bigquery upload --file_path tiffs/cartocog_byte_10km_MOD12Q1_Land_Cover_Type2_gmaps.tif --project cartodb-on-gcp-backend-team --dataset juanra --table cartocog_byte_10km_MOD12Q1_Land_Cover_Type2 --overwrite --band 1 --band 2 --token XXX

With:

    tableName: 'cartodb-on-gcp-backend-team.juanra.cartocog_byte_10km_MOD12Q1_Land_Cover_Type2',
    getFillColor: (d) => {
      const { band_1, band_2 } = d.properties;
      return band_1 === 0
        ? [0, 0, 0, 100]
        : band_1 > 100
        ? [0, 0, 0, 100]
        : [0, band_1 * 10, 0];
    },

cartocog_byte_76m_20240216_N700Coverage_NSA.tif (13.7MB)

    gdalwarp -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 \
    tiffs/cartocog_byte_76m_20240216_N700Coverage_NSA.tif \
    tiffs/cartocog_byte_76m_20240216_N700Coverage_NSA_gmaps.tif

    carto bigquery upload --file_path tiffs/cartocog_byte_76m_20240216_N700Coverage_NSA_gmaps.tif --project cartodb-on-gcp-backend-team --dataset juanra --table cartocog_byte_76m_20240216_N700Coverage_NSA --overwrite --band 1 --band 2 --token ya29.a0AXooCgvBTtor41F7PDbRXBV8VoAC5KFfk1hQ_nVSERgiTo-4NTGurckXpQuHci8P_W1c15o-UiAI5dXI0GcVp-n-5P-7sTdCPCfNybNsFpovzkWKdO83F5kXanICz4_-6Ekkx2Q4z0cTsi3NQRRaeEkSPcC7DiE6jxcHrQewt3ITaCgYKAWUSARESFQHGX2MiSfepu4NfAwQv2yFQVizkBQ0179

With:

    tableName: 'cartodb-on-gcp-backend-team.juanra.cartocog_byte_10km_MOD12Q1_Land_Cover_Type2',
    getFillColor: (d) => {
      const { band_1, band_2 } = d.properties;
      return [band_1, band_1, band_1, band_2 || band_1 === 0 ? 100 : 0];
    },

lonlat_wgs84_float32_28m_borneo.tif

    gdalwarp -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 \
    tiffs/lonlat_wgs84_float32_28m_borneo.tif \
    tiffs/lonlat_wgs84_float32_28m_borneo_gmaps.tif

    carto bigquery upload --file_path tiffs/lonlat_wgs84_float32_28m_borneo_gmaps.tif --project cartodb-on-gcp-backend-team --dataset juanra --table lonlat_wgs84_float32_28m_borneo --overwrite --band 1 --band 2 --token XXX

With:

    getFillColor: (d) => {
      const { band_1, band_2 } = d.properties;
      return [band_1, band_1, band_1, band_2 || band_1 === 0 ? 100 : 0];
    },

CartoDB / raster-loader