search

stac-utils / stac-fastapi

STAC API implementation with FastAPI.
https://stac-utils.github.io/stac-fastapi/
MIT License
245 stars 99 forks source link

Issues with Search datetime parameter parsing - blank start or end range and lowercase t or z #161

Closed philvarner closed 2 years ago

philvarner commented 3 years ago

These issues were found on https://planetarycomputer.microsoft.com/api/stac/v1/ and locally running both the pgstac and sqlalchemy backends.

These datetime parameter values applied to /search should be accepted, but incorrectly return a 400:

Search with datetime=/1985-04-12T23:20:50.52Z returned status code 400
Search with datetime=1985-04-12T23:20:50.52Z/ returned status code 400

This is likely because only .. is accepted as an open ended interval, whereas blank string is also allowed per: "Open ranges in time intervals at the start or end are supported using a double-dot (..) or an empty string for the start/end." http://docs.opengeospatial.org/is/17-069r3/17-069r3.html#_parameter_datetime

This datetime string gave a 400, likely because it uses t and z, which is acceptable.

Search with datetime=1985-04-12t23:20:50.000z returned status code 400

A large number of valid RFC3339 strings can be found here: https://github.com/stac-utils/stac-api-validator/blob/afb5a91ed6ce804d6b33fb304d6532dd4c50a227/stac_api_validator/validations.py#L38

geospatial-jeff commented 3 years ago

The new version of stac-pydantic will use pydantic's datetime parser, which still doesn't quite do the job:

from pydantic.datetime_parse import parse_datetime

dates = [
    "1985-04-12T23:20:50.52Z",
    "1985-04-12",
    "1937-01-01T12:00:27.87+0100",
    "37-01-01T12:00:27.87",
    "1985-12-12T23:20:50.52",
    "1985-04-12T23:20:50,Z"
]

for d in dates:
    try:
        parse_datetime(d)
    except:
        print("Invalid: ", d)
        continue
    print("Valid: ", d)

"Valid:  1985-04-12T23:20:50.52Z"
"Invalid:  1985-04-12" # date only
"Valid:  1937-01-01T12:00:27.87+0100" # timezone offset must have colon
"Invalid:  37-01-01T12:00:27.87" # 2 digit years not accepted
"Valid:  1985-12-12T23:20:50.52" # datetimes must have a timezone
"Invalid:  1985-04-12T23:20:50,Z" # must be at least one digit of fractional seconds

Looks like most of the cases that break are related to the timezone requirements:

  • a timezone offset must have a colon between the HH and MM
  • datetimes must have a timezone (Z or [+-]HH:MM)

I've been thinking about writing a small and generic python library for validating datetime objects against the stac spec, as none of the existing implementations (pydantic, dateutil, rfc3339-validator) seem to capture all of the edge cases.

philvarner commented 3 years ago

If you do, I have more details and what I think it a correct regex here https://github.com/radiantearth/stac-api-spec/pull/162

philvarner commented 2 years ago

@geospatial-jeff I have a section in the STAC API spec now https://github.com/radiantearth/stac-api-spec/blob/master/implementation.md#datetime-parameter-handling

https://github.com/closeio/ciso8601 has a correct RFC3339 parser.

geospatial-jeff commented 2 years ago

closed with #368