search

developmentseed / eoAPI

[Active Development] Earth Observation API (Metadata, Raster and Vector services)
https://eoapi.dev
MIT License
202 stars 21 forks source link

Invalid Assets in stac.eoapi #185

Closed vincentsarago closed 5 months ago

vincentsarago commented 5 months ago

https://stac.eoapi.dev/collections/MAXAR_Emilia_Romagna_Italy_flooding_may23/items/33_031111212102_10300500DDF89E00

in stac.eoapi we ingested Items and Collection from https://github.com/vincentsarago/MAXAR_opendata_to_pgstac

I'm not sure what is going on but the STAC API adds 

        "cloud-mask": {
            "type": "application/geopackage+sqlite3",
            "roles": [
                "cloud"
            ],
            "title": "Cloud Polygons"
        },

to the /items/{item_id} response.

I'm pretty sure this are added (hydrated) from the collection's item-assets 

{
    "type": "Collection",
    "id": "MAXAR_Emilia_Romagna_Italy_flooding_may23",
    ... 
    "item_assets": {
        ...
        "cloud-mask": {
            "type": "application/geopackage+sqlite3",
            "title": "Cloud Polygons",
            "roles": [
                "cloud"
            ]
        },
        ...

}

cc @bitner

bitner commented 5 months ago

That is almost certainly an issue with hydration in pgstac. Can you post the full collection json and the original json for the item so I can try to recreate?

vincentsarago commented 5 months ago

confirmed that there is something happening withing pgstac

  1. Select from items: no cloud-mask assets
$  SELECT content from items WHERE id='33_031111212102_10300500DDF89E00'

{
    "bbox": [
        12.237464282342282,
        44.351592,
        12.283876,
        44.35738018889801
    ],
    "links": [],
    "assets": {
        "visual": {
            "href": "s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-visual.tif",
            "eo:bands": [
                {
                    "name": "BAND_R",
                    "common_name": "red",
                    "description": "Red"
                },
                {
                    "name": "BAND_G",
                    "common_name": "green",
                    "description": "Green"
                },
                {
                    "name": "BAND_B",
                    "common_name": "blue",
                    "description": "Blue"
                }
            ],
            "alternate": {
                "public": {
                    "href": "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-visual.tif",
                    "title": "Public Access"
                }
            },
            "proj:bbox": [
                279843.75,
                4909843.75,
                285156.25,
                4915156.25
            ],
            "proj:shape": [
                17408,
                17408
            ],
            "proj:transform": [
                0.30517578125,
                0,
                279843.75,
                0,
                -0.30517578125,
                4915156.25,
                0,
                0,
                1
            ]
        },
        "data-mask": {
            "href": "s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-data-mask.gpkg",
            "alternate": {
                "public": {
                    "href": "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-data-mask.gpkg",
                    "title": "Public Access"
                }
            }
        },
        "ms_analytic": {
            "href": "s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-ms.tif",
            "eo:bands": [
                {
                    "name": "BAND_C",
                    "common_name": "coastal",
                    "description": "Coastal Blue"
                },
                {
                    "name": "BAND_B",
                    "common_name": "blue",
                    "description": "Blue"
                },
                {
                    "name": "BAND_G",
                    "common_name": "green",
                    "description": "Green"
                },
                {
                    "name": "BAND_Y",
                    "common_name": "yellow",
                    "description": "Yellow"
                },
                {
                    "name": "BAND_R",
                    "common_name": "red",
                    "description": "Red"
                },
                {
                    "name": "BAND_RE",
                    "common_name": "rededge",
                    "description": "Red Edge 1"
                },
                {
                    "name": "BAND_N",
                    "common_name": "nir08",
                    "description": "Near Infrared 1"
                },
                {
                    "name": "BAND_N2",
                    "common_name": "nir09",
                    "description": "Near Infrared 2"
                }
            ],
            "alternate": {
                "public": {
                    "href": "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-ms.tif",
                    "title": "Public Access"
                }
            },
            "proj:bbox": [
                279843.75,
                4909843.75,
                285156.25,
                4915156.25
            ],
            "proj:shape": [
                2509,
                2509
            ],
            "proj:transform": [
                2.117377441211638,
                0,
                279843.75,
                0,
                -2.117377441211638,
                4915156.25,
                0,
                0,
                1
            ]
        },
        "pan_analytic": {
            "href": "s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-pan.tif",
            "eo:bands": [
                {
                    "name": "BAND_P",
                    "description": "Pan"
                }
            ],
            "alternate": {
                "public": {
                    "href": "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-pan.tif",
                    "title": "Public Access"
                }
            },
            "proj:bbox": [
                279843.75,
                4909843.75,
                285156.25,
                4915156.25
            ],
            "proj:shape": [
                10036,
                10036
            ],
            "proj:transform": [
                0.5293443603029095,
                0,
                279843.75,
                0,
                -0.5293443603029095,
                4915156.25,
                0,
                0,
                1
            ]
        }
    },
    "properties": {
        "gsd": 0.54,
        "quadkey": "031111212102",
        "datetime": "2023-04-23T10:32:10Z",
        "platform": "WV02",
        "utm_zone": 33,
        "grid:code": "MXRA-Z33-031111212102",
        "proj:bbox": [
            279843.75,
            4914512.855107463,
            283546.65504991356,
            4915156.25
        ],
        "proj:epsg": 32633,
        "catalog_id": "10300500DDF89E00",
        "view:azimuth": 305.3,
        "proj:geometry": {
            "type": "Polygon",
            "coordinates": [
                [
                    [
                        279843.75,
                        4915156.25
                    ],
                    [
                        283546.65504991356,
                        4915156.25
                    ],
                    [
                        283525.29480963177,
                        4914512.855107463
                    ],
                    [
                        279843.75,
                        4914638.351984056
                    ],
                    [
                        279843.75,
                        4915156.25
                    ]
                ]
            ]
        },
        "tile:data_area": 2.1,
        "view:off_nadir": 23.5,
        "tile:clouds_area": 0,
        "view:sun_azimuth": 162.9,
        "view:sun_elevation": 57.3,
        "tile:clouds_percent": 0,
        "ard_metadata_version": "0.0.1",
        "view:incidence_angle": 63.4
    },
    "stac_extensions": [
        "https://stac-extensions.github.io/view/v1.0.0/schema.json",
        "https://stac-extensions.github.io/projection/v1.0.0/schema.json",
        "https://stac-extensions.github.io/eo/v1.0.0/schema.json",
        "https://stac-extensions.github.io/raster/v1.1.0/schema.json",
        "https://stac-extensions.github.io/grid/v1.0.0/schema.json",
        "https://stac-extensions.github.io/alternate-assets/v1.1.0/schema.json"
    ]
}
  1. Select * FROM pgstac.search: cloud-mask assets
FROM pgstac.search('{"colllections":["MAXAR_Emilia_Romagna_Italy_flooding_may23"],"ids":["33_031111212102_10300500DDF89E00"]}');

NOTICE:  Checking if update is needed for: id = ANY ('{33_031111212102_10300500DDF89E00}')  .
NOTICE:  Stats Last Updated: 2024-04-10 14:06:06.885293+00
NOTICE:  TTL: 1 day, Age: 00:00:21.801043
NOTICE:  Context: on, Existing Total: 1
NOTICE:  Checking if update is needed for: id = ANY ('{33_031111212102_10300500DDF89E00}')  .
NOTICE:  Stats Last Updated: 2024-04-10 14:06:06.885293+00
NOTICE:  TTL: 1 day, Age: 00:00:21.801043
NOTICE:  Context: on, Existing Total: 1
NOTICE:  SEARCH:TOKEN: <NULL>
NOTICE:  ***TOKEN: <NULL>
NOTICE:  FULL WHERE CLAUSE: id = ANY ('{33_031111212102_10300500DDF89E00}') 
NOTICE:  Time to get counts and build query 2.64
NOTICE:  Getting hydrated data.
NOTICE:  CACHE SET TO f
NOTICE:  Time to set hydration/formatting 0.153
NOTICE:  Getting chunks for id = ANY ('{33_031111212102_10300500DDF89E00}')  datetime DESC, id DESC
NOTICE:  Running Query for -infinity to infinity. 4.603
NOTICE:  Returned 1/11 Rows From -infinity to infinity. 10 to go. Time: 4.038ms
NOTICE:  SEARCH_ROWS TOOK 8.772ms
NOTICE:  Time to fetch rows 20.916
NOTICE:  Query returned 1 records.
NOTICE:  Time to get prev/next 0.154
NOTICE:  Time to build final json 0.31
NOTICE:  Total Time: 24.435
NOTICE:  RETURNING 1 records. NEXT: <NULL>. PREV: <NULL>

{
    "next": null,
    "prev": null,
    "type": "FeatureCollection",
    "context": {
        "limit": 10,
        "matched": 1,
        "returned": 1
    },
    "features": [
        {
            "id": "33_031111212102_10300500DDF89E00",
            "bbox": [
                12.237464282342282,
                44.351592,
                12.283876,
                44.35738018889801
            ],
            "type": "Feature",
            "links": [],
            "assets": {
                ...
                "data-mask": {
                    "href": "s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-data-mask.gpkg",
                    "type": "application/geopackage+sqlite3",
                    "roles": [
                        "data-mask"
                    ],
                    "title": "Data Mask",
                    "alternate": {
                        "public": {
                            "href": "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-data-mask.gpkg",
                            "title": "Public Access"
                        }
                    }
                },
                "cloud-mask": {
                    "type": "application/geopackage+sqlite3",
                    "roles": [
                        "cloud"
                    ],
                    "title": "Cloud Polygons"
                },
                ...
    ]
}
vincentsarago commented 5 months ago

@bitner collections and items are from https://github.com/vincentsarago/MAXAR_opendata_to_pgstac/tree/main/Maxar

vincentsarago commented 5 months ago

from slack

so the issues are :

bitner commented 5 months ago

Quick test that shows how this should be working. This is on my dev environment, I'll check the actual case above on that database in a minute.

insert into collections (content) values ('{"id":"collection","item_assets":{"cloud-mask": {
            "type": "application/geopackage+sqlite3",
            "title": "Cloud Polygons",
            "roles": [
                "cloud"
            ]
        }}}');
postgis=# select content_slim('{"id":"myid","collection":"collection","assets":{"a":"b"}}');
                content_slim                
--------------------------------------------
 {"assets": {"a": "b", "cloud-mask": "𒍟※"}}
(1 row)

You can see that since cloud-mask was not present in the item json, when we dehydrate it, it is given the "don't hydrate marker"

Now when we hydrate...

postgis=# select content_hydrate(content_slim('{"id":"myid","collection":"collection","assets":{"a":"b"}}'), (SELECT base_item from collections where id='collection'));
                            content_hydrate                            
-----------------------------------------------------------------------
 {"type": "Feature", "assets": {"a": "b"}, "collection": "collection"}
(1 row)

... we get back the item without the cloud-mask entries

bitner commented 5 months ago

Hrmmmm....

eoapi=> select jsonb_each(content_slim('
{"type": "Feature", "stac_version": "1.0.0", "id": "33_031111212102_10300500DDF89E00", "properties": {"datetime": "2023-04-23T10:32:10Z", "platform": "WV02", "gsd": 0.54, "ard_metadata_version": "0.0.1", "catalog_id": "10300500DDF89E00", "utm_zone": 33, "quadkey": "031111212102", "view:off_nadir": 23.5, "view:azimuth": 305.3, "view:incidence_angle": 63.4, "view:sun_azimuth": 162.9, "view:sun_elevation": 57.3, "proj:epsg": 32633, "proj:geometry": {"type": "Polygon", "coordinates": [[[279843.75, 4915156.25], [283546.65504991356, 4915156.25], [283525.29480963177, 4914512.855107463], [279843.75, 4914638.351984056], [279843.75, 4915156.25]]]}, "grid:code": "MXRA-Z33-031111212102", "proj:bbox": [279843.75, 4914512.855107463, 283546.65504991356, 4915156.25], "tile:data_area": 2.1, "tile:clouds_area": 0.0, "tile:clouds_percent": 0}, "geometry": {"type": "Polygon", "coordinates": [[[12.237464282342282, 44.35627042177566], [12.283876, 44.35738018889801], [12.283876, 44.351592], [12.237684007987614, 44.351592], [12.237464282342282, 44.35627042177566]]]}, "links": [], "assets": {"data-mask": {"href": "s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-data-mask.gpkg", "type": "application/geopackage+sqlite3", "title": "Data Mask", "roles": ["data-mask"], "alternate": {"public": {"title": "Public Access", "href": "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-data-mask.gpkg"}}}, "ms_analytic": {"href": "s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-ms.tif", "type": "image/tiff; application=geotiff; profile=cloud-optimized", "title": "Multispectral Image", "proj:bbox": [279843.75, 4909843.75, 285156.25, 4915156.25], "proj:shape": [2509, 2509], "proj:transform": [2.117377441211638, 0.0, 279843.75, 0.0, -2.117377441211638, 4915156.25, 0.0, 0.0, 1.0], "eo:bands": [{"name": "BAND_C", "common_name": "coastal", "description": "Coastal Blue"}, {"name": "BAND_B", "common_name": "blue", "description": "Blue"}, {"name": "BAND_G", "common_name": "green", "description": "Green"}, {"name": "BAND_Y", "common_name": "yellow", "description": "Yellow"}, {"name": "BAND_R", "common_name": "red", "description": "Red"}, {"name": "BAND_RE", "common_name": "rededge", "description": "Red Edge 1"}, {"name": "BAND_N", "common_name": "nir08", "description": "Near Infrared 1"}, {"name": "BAND_N2", "common_name": "nir09", "description": "Near Infrared 2"}], "roles": ["data"], "alternate": {"public": {"title": "Public Access", "href": "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-ms.tif"}}}, "pan_analytic": {"href": "s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-pan.tif", "type": "image/tiff; application=geotiff; profile=cloud-optimized", "title": "Panchromatic Image", "proj:bbox": [279843.75, 4909843.75, 285156.25, 4915156.25], "proj:shape": [10036, 10036], "proj:transform": [0.5293443603029095, 0.0, 279843.75, 0.0, -0.5293443603029095, 4915156.25, 0.0, 0.0, 1.0], "eo:bands": [{"name": "BAND_P", "description": "Pan"}], "roles": ["data"], "alternate": {"public": {"title": "Public Access", "href": "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-pan.tif"}}}, "visual": {"href": "s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-visual.tif", "type": "image/tiff; application=geotiff; profile=cloud-optimized", "title": "Visual Image", "proj:bbox": [279843.75, 4909843.75, 285156.25, 4915156.25], "proj:shape": [17408, 17408], "proj:transform": [0.30517578125, 0.0, 279843.75, 0.0, -0.30517578125, 4915156.25, 0.0, 0.0, 1.0], "eo:bands": [{"name": "BAND_R", "common_name": "red", "description": "Red"}, {"name": "BAND_G", "common_name": "green", "description": "Green"}, {"name": "BAND_B", "common_name": "blue", "description": "Blue"}], "roles": ["visual"], "alternate": {"public": {"title": "Public Access", "href": "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-visual.tif"}}}}, "bbox": [12.237464282342282, 44.351592, 12.283876, 44.35738018889801], "stac_extensions": ["https://stac-extensions.github.io/view/v1.0.0/schema.json", "https://stac-extensions.github.io/projection/v1.1.0/schema.json", "https://stac-extensions.github.io/eo/v1.1.0/schema.json", "https://stac-extensions.github.io/raster/v1.1.0/schema.json", "https://stac-extensions.github.io/grid/v1.1.0/schema.json", "https://stac-extensions.github.io/alternate-assets/v1.1.0/schema.json"], "collection": "MAXAR_Emilia_Romagna_Italy_flooding_may23"}
'::jsonb)->'assets')
;
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               jsonb_each                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 (visual,"{""href"": ""s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-visual.tif"", ""eo:bands"": [{""name"": ""BAND_R"", ""common_name"": ""red"", ""description"": ""Red""}, {""name"": ""BAND_G"", ""common_name"": ""green"", ""description"": ""Green""}, {""name"": ""BAND_B"", ""common_name"": ""blue"", ""description"": ""Blue""}], ""alternate"": {""public"": {""href"": ""https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-visual.tif"", ""title"": ""Public Access""}}, ""proj:bbox"": [279843.75, 4909843.75, 285156.25, 4915156.25], ""proj:shape"": [17408, 17408], ""proj:transform"": [0.30517578125, 0.0, 279843.75, 0.0, -0.30517578125, 4915156.25, 0.0, 0.0, 1.0]}")
 (data-mask,"{""href"": ""s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-data-mask.gpkg"", ""alternate"": {""public"": {""href"": ""https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-data-mask.gpkg"", ""title"": ""Public Access""}}}")
 (cloud-mask,"""𒍟※""")
 (ms_analytic,"{""href"": ""s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-ms.tif"", ""eo:bands"": [{""name"": ""BAND_C"", ""common_name"": ""coastal"", ""description"": ""Coastal Blue""}, {""name"": ""BAND_B"", ""common_name"": ""blue"", ""description"": ""Blue""}, {""name"": ""BAND_G"", ""common_name"": ""green"", ""description"": ""Green""}, {""name"": ""BAND_Y"", ""common_name"": ""yellow"", ""description"": ""Yellow""}, {""name"": ""BAND_R"", ""common_name"": ""red"", ""description"": ""Red""}, {""name"": ""BAND_RE"", ""common_name"": ""rededge"", ""description"": ""Red Edge 1""}, {""name"": ""BAND_N"", ""common_name"": ""nir08"", ""description"": ""Near Infrared 1""}, {""name"": ""BAND_N2"", ""common_name"": ""nir09"", ""description"": ""Near Infrared 2""}], ""alternate"": {""public"": {""href"": ""https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-ms.tif"", ""title"": ""Public Access""}}, ""proj:bbox"": [279843.75, 4909843.75, 285156.25, 4915156.25], ""proj:shape"": [2509, 2509], ""proj:transform"": [2.117377441211638, 0.0, 279843.75, 0.0, -2.117377441211638, 4915156.25, 0.0, 0.0, 1.0]}")
 (pan_analytic,"{""href"": ""s3://maxar-opendata/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-pan.tif"", ""eo:bands"": [{""name"": ""BAND_P"", ""description"": ""Pan""}], ""alternate"": {""public"": {""href"": ""https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111212102/2023-04-23/10300500DDF89E00-pan.tif"", ""title"": ""Public Access""}}, ""proj:bbox"": [279843.75, 4909843.75, 285156.25, 4915156.25], ""proj:shape"": [10036, 10036], ""proj:transform"": [0.5293443603029095, 0.0, 279843.75, 0.0, -0.5293443603029095, 4915156.25, 0.0, 0.0, 1.0]}")
 (cloud-mask-raster,"""𒍟※""")
 (cloud-shadow-mask,"""𒍟※""")
(7 rows)

This looks like when I run the dehydrate manually, all the 'do not merge' markers get filled in correctly