global data in s3 bucket not a valid Cloud Optimized GeoTiff

[h-a-graham]

Hey! Firstly, thanks so much for releasing these data and the models - it's really interesting and exciting! I have a question regarding the file format for the global data on the s3 bucket at: s3://dataforgood-fb-data/forests/v1/alsgedi_global_v6_float/chm/

It appears that the tif files are not valid Cloud Optimized Geotiffs - it would be amazing if valid COGS could be made available to improve the accessibility of these data.

example to verify COG status using riocogeo and gdalinfo:

export AWS_NO_SIGN_REQUEST=YES

path="/vsis3/dataforgood-fb-data/forests/v1/alsgedi_global_v6_float/chm/300001101.tif"

rio cogeo validate $path
gdalinfo $path

The following warnings were found:
- The file is greater than 512xH or 512xW, it is recommended to include internal overviews

The following errors were found:
- The file is greater than 512xH or 512xW, but is not tiled
/vsis3/dataforgood-fb-data/forests/v1/alsgedi_global_v6_float/chm/300001101.tif is NOT a valid cloud optimized GeoTIFF
Driver: GTiff/GeoTIFF
Files: /vsis3/dataforgood-fb-data/forests/v1/alsgedi_global_v6_float/chm/300001101.tif
Size is 65536, 65536
Coordinate System is:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
            MEMBER["World Geodetic System 1984 (G1762)"],
            MEMBER["World Geodetic System 1984 (G2139)"],
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]],
            ENSEMBLEACCURACY[2.0]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["Popular Visualisation Pseudo-Mercator",
        METHOD["Popular Visualisation Pseudo Mercator",
            ID["EPSG",1024]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting (X)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing (Y)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Web mapping and visualisation."],
        AREA["World between 85.06°S and 85.06°N."],
        BBOX[-85.06,-180,85.06,180]],
    ID["EPSG",3857]]
Data axis to CRS axis mapping: 1,2
Origin = (1017529.720532266306691,0.000000000000000)
Pixel Size = (1.194328566955877,-1.194328566955879)
Metadata:
  AREA_OR_POINT=Area
  UNITS=METERS
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
  PREDICTOR=2
Corner Coordinates:
Upper Left  ( 1017529.721,       0.000) (  9d 8'26.25"E,  0d 0' 0.01"N)
Lower Left  ( 1017529.721,  -78271.517) (  9d 8'26.25"E,  0d42'11.19"S)
Upper Right ( 1095801.237,       0.000) (  9d50'37.50"E,  0d 0' 0.01"N)
Lower Right ( 1095801.237,  -78271.517) (  9d50'37.50"E,  0d42'11.19"S)
Center      ( 1056665.479,  -39135.758) (  9d29'31.87"E,  0d21' 5.62"S)
Band 1 Block=65536x1 Type=Byte, ColorInterp=Gray

And also I saw somewhere that an "image date" layer had been made available on earth engine, are there any plans to also include this layer in the s3 bucket?

Many thanks, Hugh

[coupriec]

Thank you for pointing that out, we will look into this. About the image date layer, we will upload a map of the dates on s3 very soon.

[h-a-graham]

Much appreciated, I look forward to hearing from you! Thanks

[coupriec]

Hi, the observation dates are available in three forms:

• An S3 bucket containing geojsons specifying the observation date. The geojson filenames correspond to the quadkey for each zoom=9 tile, which is the same filenames that the geotiffs use. The S3 uri is: “s3://dataforgood-fb-data/forests/v1/alsgedi_global_v6_float/metadata/”
• We also have a raster file in S3 that specifies observation date. The units are years since 2000: s3://dataforgood-fb-data/forests/v1/alsgedi_global_v6_float/CHM_acquisition_date.tif
• The observation date is also available as a layer in the Google Earth Engine interface. The units are years since 2000. Making Cloud Optimized Geotiffs is a bit more complex, I can't give a timeline for that.

[h-a-graham]

Many thanks - that's very helpful and I understand that reproducing as COGs will be a bit of process - cheers!