Closed pedro-cf closed 6 months ago
I feel like this is a bug but I'm not entirely sure - the best might be to run stac-fastapi-elasticsearch
I feel like this is a bug but I'm not entirely sure - the best might be to run stac-fastapi-elasticsearch
Confirmed. Works just fine in https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch
Output from sfeo POST http://localhost:8080/collections/test-collection/items
:
{
"type": "Feature",
"stac_version": "1.0.0",
"stac_extensions": [
"https://stac-extensions.github.io/web-map-links/v1.2.0/schema.json"
],
"id": "item",
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
172.9,
1.3
],
[
173,
1.3
],
[
173,
1.4
],
[
172.9,
1.4
],
[
172.9,
1.3
]
]
]
},
"bbox": [
172.9,
1.3,
173.0,
1.4
],
"properties": {
"datetime": "2020-12-11T22:38:32Z",
"created": "2024-04-24T16:21:54.865138Z",
"updated": "2024-04-24T16:21:54.865138Z"
},
"links": [
{
"href": "https://maps.example.com/wmts",
"rel": "wmts",
"type": "image/png",
"title": "RGB composite visualized through a WMTS",
"wmts:layer": [
"streets",
"satellite"
]
},
{
"href": "https://maps.example.com/wms",
"rel": "wms",
"type": "image/png",
"title": "RGB composite visualized through a WMS",
"wms:layers": [
"rgb"
],
"wms:transparent": true
},
{
"href": "https://maps.example.com/xyz/{z}/{x}/{y}.jpg",
"rel": "xyz",
"type": "image/jpeg",
"title": "RGB composite visualized through a XYZ"
},
{
"href": "https://maps.example.com/item/tilejson.json",
"rel": "tilejson",
"title": "TileJSON",
"type": "application/json"
},
{
"href": "https://storage.googleapis.com/open-cogs/planet-stac/cocabamba-peru/3d-geofox.ai/3DTiles/tileset.json",
"rel": "3d-tiles",
"title": "3D Tiles",
"type": "application/json"
},
{
"href": "https://maps.example.com/item/example.pmtiles",
"rel": "pmtiles",
"title": "PMTiles",
"type": "application/vnd.pmtiles",
"pmtiles:layers": [
"streets"
]
}
],
"assets": {},
"collection": "test-collection"
}
Any idea what might be causing it? @jonhealy1
I will have a look. stac-fastapi-mongo uses the core library from sfeos. sfeos is much more developed.
@jonhealy1 I tried upgrading stac-fastapi.core
2.1.0
-> 2.2.0
: fixes the issue and tests are 100% passing:
2.1.0
-> 2.3.0
: fixes the issue but causes 9 test errors:
Would you say it is safe to upgrade to 2.2.0
?
If there are no errors it's good. Just make a pr to upgrade to v2.2.0. I will then relase a new version of stac-fastapi-mongo
I would like to upgrade the stac-fastapi.core
to 2.3.0
I've made this fork/branch: https://github.com/pedro-cf/stac-fastapi-mongo/tree/stac-fastapi.core-2.3.0
I've fixed 5 of the 9 tests. The errors were mostly due to this big change inte stac-fastapi.core:
PUT /collections
is now PUT /collections/{collection_id}
I've also added this to the basic_auth
tests to prevent awkawardness when calling tests outside of the github workflow, like with make test
:
if not os.getenv("BASIC_AUTH"):
pytest.skip()
I can't seem to understand how to fix these remaining tests:
When running stac_fastapi/tests/api/test_api.py::test_datetime_non_interval
:
https://github.com/Healy-Hyperspatial/stac-fastapi-mongo/blob/5bcac32d73dcda4049ee95b4fc493267e4d1ea64/stac_fastapi/tests/api/test_api.py#L399-L417
resp_json
has no features and fails with IndexError: list index out of range
When I replicate this test in a real environment I don't run into this error.
I don't really understand the intricacies of conftest.py
and ctx
and app_client
.
Maybe you can help me on understanding why app_client
and app_client.post("/search", json=params)
is acting differently?
The fourth test is a test we were struggling with that has now been fixed in stac-fastapi but the changes are not in the core library yet. There is an issue here that I made: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/issues/230
It's interesting that all of the tests failing are datetime associated
It's also weird that they weren't failing before and the same tests pass in sfeos without any changes
I'm starting it now going to see what I can figure out.
@jonhealy1 I think it's something to the with the test_item.json
I tried creating the item with the file itself:
curl --request POST \
--url http://localhost:8084/collections/test-collection/items \
--header 'Content-Type: application/json' \
--data @./stac_fastapi/tests/data/test_item.json
and I tried creating the item by copy pasting the output from resp = await app_client.post("/search", json={})
in the tests...
and when I take what's stored in the database and json diffcheck the two:
This curl is suppost to create the same item in the DB, but the datetime isn't being converted to the Mongo date format, it's say the same 2020-02-12T12:30:22+00:00
:
curl --request POST --url http://localhost:8084/collections/test-collection/items --header 'Content-Type: application/json' --data '{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[152.15052873427666,-33.82243006904891],[150.1000346138806,-34.257132625788756],[149.5776607193635,-32.514709769700254],[151.6262528041627,-32.08081674221862],[152.15052873427666,-33.82243006904891]]]},"properties":{"datetime":"2020-02-12T12:30:22+00:00"},"id":"test-item","bbox":[149.57574,-34.25796,152.15194,-32.07915],"stac_version":"1.0.0","assets":{"ANG":{"title":"Angle Coefficients File","description":"Collection 2 Level-1 Angle Coefficients File (ANG)","href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_ANG.txt","type":"text/plain"},"SR_B1":{"title":"Coastal/Aerosol Band (B1)","description":"Collection 2 Level-2 Coastal/Aerosol Band (B1) Surface Reflectance","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_SR_B1.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":30,"name":"SR_B1","common_name":"coastal","center_wavelength":0.44,"full_width_half_max":0.02}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"SR_B2":{"title":"Blue Band (B2)","description":"Collection 2 Level-2 Blue Band (B2) Surface Reflectance","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_SR_B2.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":30,"name":"SR_B2","common_name":"blue","center_wavelength":0.48,"full_width_half_max":0.06}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"SR_B3":{"title":"Green Band (B3)","description":"Collection 2 Level-2 Green Band (B3) Surface Reflectance","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_SR_B3.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":30,"name":"SR_B3","common_name":"green","center_wavelength":0.56,"full_width_half_max":0.06}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"SR_B4":{"title":"Red Band (B4)","description":"Collection 2 Level-2 Red Band (B4) Surface Reflectance","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_SR_B4.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":30,"name":"SR_B4","common_name":"red","center_wavelength":0.65,"full_width_half_max":0.04}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"SR_B5":{"title":"Near Infrared Band 0.8 (B5)","description":"Collection 2 Level-2 Near Infrared Band 0.8 (B5) Surface Reflectance","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_SR_B5.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":30,"name":"SR_B5","common_name":"nir08","center_wavelength":0.86,"full_width_half_max":0.03}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"SR_B6":{"title":"Short-wave Infrared Band 1.6 (B6)","description":"Collection 2 Level-2 Short-wave Infrared Band 1.6 (B6) Surface Reflectance","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_SR_B6.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":30,"name":"SR_B6","common_name":"swir16","center_wavelength":1.6,"full_width_half_max":0.08}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"SR_B7":{"title":"Short-wave Infrared Band 2.2 (B7)","description":"Collection 2 Level-2 Short-wave Infrared Band 2.2 (B7) Surface Reflectance","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_SR_B7.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":30,"name":"SR_B7","common_name":"swir22","center_wavelength":2.2,"full_width_half_max":0.2}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"ST_QA":{"title":"Surface Temperature Quality Assessment Band","description":"Landsat Collection 2 Level-2 Surface Temperature Band Surface Temperature Product","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_ST_QA.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"ST_B10":{"title":"Surface Temperature Band (B10)","description":"Landsat Collection 2 Level-2 Surface Temperature Band (B10) Surface Temperature Product","gsd":100,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_ST_B10.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":100,"name":"ST_B10","common_name":"lwir11","center_wavelength":10.9,"full_width_half_max":0.8}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"MTL.txt":{"title":"Product Metadata File","description":"Collection 2 Level-1 Product Metadata File (MTL)","href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_MTL.txt","type":"text/plain"},"MTL.xml":{"title":"Product Metadata File (xml)","description":"Collection 2 Level-1 Product Metadata File (xml)","href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_MTL.xml","type":"application/xml"},"ST_DRAD":{"title":"Downwelled Radiance Band","description":"Landsat Collection 2 Level-2 Downwelled Radiance Band Surface Temperature Product","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_ST_DRAD.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":30,"name":"ST_DRAD","description":"downwelled radiance"}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"ST_EMIS":{"title":"Emissivity Band","description":"Landsat Collection 2 Level-2 Emissivity Band Surface Temperature Product","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_ST_EMIS.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":30,"name":"ST_EMIS","description":"emissivity"}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]},"ST_EMSD":{"title":"Emissivity Standard Deviation Band","description":"Landsat Collection 2 Level-2 Emissivity Standard Deviation Band Surface Temperature Product","gsd":30,"href":"https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/108/066/LC08_L2SP_108066_20210712_20210720_02_T1/LC08_L2SP_108066_20210712_20210720_02_T1_ST_EMSD.TIF","type":"image/tiff; application=geotiff; profile=cloud-optimized","eo:bands":[{"gsd":30,"name":"ST_EMSD","description":"emissivity standard deviation"}],"proj:shape":[7731,7591],"proj:transform":[30,0,304185,0,-30,-843585]}},"links":[{"href":"http://test-server/collections/test-collection/items/test-item","rel":"self","type":"application/geo+json"},{"href":"http://test-server/collections/test-collection","rel":"parent","type":"application/json"},{"href":"http://test-server/collections/test-collection","rel":"collection","type":"application/json"},{"href":"http://test-server/","rel":"root","type":"application/json"}],"collection":"test-collection"}'
Is it something to do with how mongo stores the date?
The datetimes are being stored as strings in the database not as ISODate
and the /search mechanism seems to be working with strings aswell.
The test_item.json
creates an item in the db with the following format:
The stac_fastapi/tests/api/test_api.py::test_datetime_non_interval
test runs the following first query:
Query: {'$and': [{'collection': {'$in': ['test-collection']}}, {'properties.datetime': {'$gte': '2020-02-12T12:30:22+00:00'}}, {'properties.datetime': {'$lte': '2020-02-12T12:30:22+00:00'}}]}
As you see the date formats are different, and since the comparison is being done bettween date strings with different formats it doesn't work so the result is 0 features.
I hacked this repo together pretty quickly. The previous releases are here and they are probably generally better. I wanted to show that we could build another backend from the sfeos core library.
I hacked this repo together pretty quickly. The previous releases are here and they are probably generally better. I wanted to show that we could build another backend from the sfeos core library.
But it's ok to focus on this repo and try to improve right ?
How does SFEOs handle different datetime formats and does it store datetimes in a generic format ? I don't know much about elasticsearch...
Do you think the datetimes should be stored as ISODates in MongoDB?
if yes:
Do we just try to parse every field with dateutil.parser
perhaps ? or just specific fields (datetime,start_datetime,end_datetime,created,updated)?
Do we enforce ISODate formats only ?
I do like mongo a lot and I have used mongo for clients when I had the choice at my old job. I think the project is worth improving but almost everyone I have talked to in geospatial seems to have an opposite opinion. I was working on stac fastapi mongo first but no one was paying attention then I did elasticsearch and people were like wow - elasticsearch.
I do like mongo a lot and I have used mongo for clients when I had the choice at my old job. I think the project is worth improving but almost everyone I have talked to in geospatial seems to have an opposite opinion. I was working on stac fastapi mongo first but no one was paying attention then I did elasticsearch and people were like wow - elasticsearch.
I think mongo has it's advantages, it's cheap, easy to setup managed services in the cloud and works well on-premises.
I attended a conference and Paul Ramsey, the guy who did A LOT of the geospatial stuff for postgis, asked the creator of stac-fastapi-pgstac how come they didn't build it in mongo. I thought that was pretty cool. The datetime stuff really confuses me. I haven't worked with it a lot. Everything I have done is just: 2020-03-01T00:00:00Z
I didn't know anything about es when I started stac-fatsapi-es. Phil Varner stepped in to help out with that part later on behalf of a big company because they were wondering about the performance implications
I still don't know that much about es or mongo ....
I think the big issue is a managed service for elasticsearch is very expensive on Azure/AWS and I'm not even sure if it would connect seamlessly with the sfeos-app. Have u tried deploying sfeos on azure f.e. connected with a Azure Elastic Search managed service?
There is opensearch I guess. I had always used the free tier on mongo atlas but then recently we moved a service to aws documentdb which is compatible. I have never had the opportunity to run stac-fastapi-es in the cloud. I'm pretty sure it would be fine.
Honestly it may be more productive for you to look at stac-fastapi-es. You should get more attention for your contributions. You are a very good programmer. I know it doesn't have a lot of stars etc. but I know of a few very big companies who use it or who have considered it. It would be good to move the basic auth stuff there for example. I haven't used Azure much - a little in school. I'm definitely down for continuing to work on this project don't get me wrong.
mongo atlas has serverless options which aws documentdb doesn't have. I wanted to use it for a non-profit we were working for but our pm thought otherwise.
One of the many things changed in from 2.2.0 to 2.3.0:
POST /search
{
"collections": [
"test-collection"
],
"datetime": "2020-02-12T12:30:22Z/.."
}
Query in 2.2.0
:
{'$and': [{'collection': {'$in': ['test-collection']}}, {'properties.datetime': {'$gte': '2020-02-12T12:30:22Z'}}, {'properties.datetime': {'$lte': '2200-12-01T12:31:12Z'}}]}
Query in 2.3.0
:
{'$and': [{'collection': {'$in': ['test-collection']}}, {'properties.datetime': {'$gte': '2020-02-12T12:30:22.000Z'}}, {'properties.datetime': {'$lte': None}}]}
this query won't work with pymongo
Is your feature request related to a problem? Please describe. Greetings I was doing some tests and trying to create an item that included links with this schema for WMS/WMTS/etc services (https://github.com/stac-extensions/web-map-links):
Here is the actual item created in the database after a
POST /collections/test-collection/items
:It seems these links are being ignored? I'm a bit lost. Do I need to add a conformance class (don't know how) or does this require addition development ? or is it a bug?
@jonhealy1