microsoft / PlanetaryComputer

Issues, discussions, and information about the Microsoft Planetary Computer
https://planetarycomputer.microsoft.com/
MIT License
185 stars 9 forks source link

sortby cloud cover, item order incorrect #81

Closed mhungen closed 2 years ago

mhungen commented 2 years ago

When trying to sort an item search by "eo:cloud_cover", I receive a list that is only partly ordered.

For example, the following query contains an element that does not match the expected result: search via GET

result:

{
    "type": "FeatureCollection",
    "features": [
        {
            "id": "S2B_MSIL2A_20220222T101019_R022_T33UVP_20220228T142848",
            "collection": "sentinel-2-l2a",
            "properties": {
                "eo:cloud_cover": 89.252603,
            },
        },
        {
            "id": "S2A_MSIL2A_20220309T100841_R022_T33UVP_20220309T204833",
            "collection": "sentinel-2-l2a",
            "properties": {
                "eo:cloud_cover": 9.008443,
            },
        },
        {
            "id": "S2A_MSIL2A_20220408T100601_R022_T33UVP_20220408T223422",
            "collection": "sentinel-2-l2a",
            "properties": {
                "eo:cloud_cover": 94.83971,
            },
        }
    ]
}

result as textfile

TomAugspurger commented 2 years ago

Thanks for the report. Here's the output I see

```python [0.131671, 0.32598, 0.55309, 1.263998, 13.975669, 17.852251, 18.132679, 22.361308, 35.051498, 35.793611, 43.404394, 50.785601, 52.026653, 62.783337, 68.834466, 89.192712, 89.252603, 9.008443, 94.83971, 99.515051, 99.549466, 99.678284, 99.776077, 99.916273] ```

In particular, note the subsequence [89.252603, 9.008443, 94.83971]. It almost looks like we're sorting a stringified version of the value. @mmcfarland, do you think this is more likely a bug in pgstac's sorting, or a bug in how we're loading the data?

mmcfarland commented 2 years ago

Yes, thanks for the report. I've been able to track this down to a bug in pgstac which is incorrectly casting the provided sort field to text within the generated sql order by clause. Looking into a fix now and will provide a link to the issue when I file it on the upstream repo.

mmcfarland commented 2 years ago

FYI, upstream issue is at https://github.com/stac-utils/pgstac/issues/134. It's not specifically a sort issue, but some inconsistencies in schema definition can result in this type of behavior for search or sort. Note that this area of pgstac is under active development: https://github.com/stac-utils/pgstac/pull/133.

As you can see in the issue, omitting the properties. in the sort field definition should return the correct result if you're looking for an immediate workaround, but the syntax you have is valid and we'll get that fixed.

TomAugspurger commented 2 years ago

This is fixed upstream. Closing.