search

cogeotiff / rio-tiler

User friendly Rasterio plugin to read raster datasets.
https://cogeotiff.github.io/rio-tiler/
BSD 3-Clause "New" or "Revised" License
502 stars 106 forks source link

fetch in stac.py read data: URLs? #408

Closed DanielJDufour closed 3 years ago

DanielJDufour commented 3 years ago

Hello. I was wondering if you would be open to a pull request adding support to the fetch function in stac.py for data: URLs. The use case is I would like a to be able to pass STAC data directly to the /stac/... endpoints in titiler without having to send a normal URL. And I believe titiler uses this library for this use case. Happy to discuss more and apologies if I'm mistaken.

Here would be an example URL:

data:application/json;base64,eyJ0eXBlIjoiRmVhdHVyZSIsInN0YWNfdmVyc2lvbiI6IjEuMC4wLWJldGEuMiIsImlkIjoiMjAxNzA4MzFfMTcyNzU0XzEwMWMiLCJwcm9wZXJ0aWVzIjp7ImRhdGV0aW1lIjoiMjAxNy0wOC0zMVQxNzoyNzo1NC45NjA1MzBaIiwiaW5zdHJ1bWVudHMiOlsiUFMyIl0sInBsYXRmb3JtIjoiMTAxYyIsImNvbnN0ZWxsYXRpb24iOiJwbGFuZXRzY29wZSIsImNyZWF0ZWQiOiIyMDE3LTEwLTI3VDEyOjE0OjA3WiIsInVwZGF0ZWQiOiIyMDE3LTEyLTA3VDA3OjM3OjI4WiIsImdzZCI6Mi42LCJlbzpjbG91ZF9jb3ZlciI6MiwidmlldzpzdW5fYXppbXV0aCI6MTQ1LjQsInZpZXc6c3VuX2VsZXZhdGlvbiI6NjUuMSwidmlldzpvZmZfbmFkaXIiOjAuMiwicHJvajplcHNnIjozMjYxNSwicGw6YW5vbWFsb3VzX3BpeGVscyI6MC4wMSwicGw6Y29sdW1ucyI6ODMwNywicGw6Z3JvdW5kX2NvbnRyb2wiOnRydWUsInBsOml0ZW1fdHlwZSI6IlBTU2NlbmU0QmFuZCIsInBsOm9yaWdpbl94IjoyMDU1MDMsInBsOm9yaWdpbl95IjozMjgwMzIwLCJwbDpwaXhlbF9yZXNvbHV0aW9uIjozLCJwbDpxdWFsaXR5X2NhdGVnb3J5IjoidGVzdCIsInBsOnJvd3MiOjM5MTgsInBsOnN0cmlwX2lkIjoiNzI0NTUzIiwicGw6dXNhYmxlX2RhdGEiOjB9LCJnZW9tZXRyeSI6eyJ0eXBlIjoiUG9seWdvbiIsImNvb3JkaW5hdGVzIjpbW1stOTYuMDMzNTI3NDY0NTg5NjIsMjkuNTQ5MDU3OTM5ODg4NTg0XSxbLTk2LjAzODEwMDA2ODQyMzMzLDI5LjU2OTQ1NDIyNzA4NjU1M10sWy05Ni4wMzczMDc5NzA3NjU4MiwyOS41Njk1OTE1NDM1NTM5MzNdLFstOTYuMDQwMDE2MjM4MDk5MTMsMjkuNTgxMjkwNzc1MTE3NDczXSxbLTk1Ljc5NzgxNzIwMDM3NTc3LDI5LjYyMzMwODE3MDYwNTE4XSxbLTk1Ljc4NjcyMTYxNzc5MjM0LDI5LjU3NTA5NjEzNzA4NzA3M10sWy05NS43ODIxMzgyNjUyODkzLDI5LjU1NDQ3NDI0NjUxODE4XSxbLTk2LjAyNTAzNzYwMTc0MTgsMjkuNTEyMzQ0MzU5NTc2MTA1XSxbLTk2LjAzMzUyNzQ2NDU4OTYyLDI5LjU0OTA1NzkzOTg4ODU4NF1dXX0sImxpbmtzIjpbeyJyZWwiOiJjb2xsZWN0aW9uIiwiaHJlZiI6Ii4uLy4uLy4uL2NvbGxlY3Rpb24uanNvbiJ9LHsicmVsIjoicm9vdCIsImhyZWYiOiIuLi8uLi8uLi9jb2xsZWN0aW9uLmpzb24iLCJ0eXBlIjoiYXBwbGljYXRpb24vanNvbiJ9LHsicmVsIjoicGFyZW50IiwiaHJlZiI6Ii4uL2NhdGFsb2cuanNvbiIsInR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIn1dLCJhc3NldHMiOnsidGh1bWJuYWlsIjp7ImhyZWYiOiJodHRwczovL3N0b3JhZ2UuZ29vZ2xlYXBpcy5jb20vcGRkLXN0YWMvZGlzYXN0ZXJzL2h1cnJpY2FuZS1oYXJ2ZXkvMDgzMS8yMDE3MDgzMV8xNzI3NTRfMTAxY190aHVtYl9sYXJnZS5wbmciLCJ0eXBlIjoiaW1hZ2UvcG5nIiwidGl0bGUiOiJUaHVtYm5haWwiLCJyb2xlcyI6WyJ0aHVtYm5haWwiXX0sImFuYWx5dGljIjp7ImhyZWYiOiJodHRwczovL3N0b3JhZ2UuZ29vZ2xlYXBpcy5jb20vcGRkLXN0YWMvZGlzYXN0ZXJzL2h1cnJpY2FuZS1oYXJ2ZXkvMDgzMS8yMDE3MDgzMV8xNzI3NTRfMTAxY18zQl9BbmFseXRpY01TLnRpZiIsInR5cGUiOiJpbWFnZS92bmQuc3RhYy5nZW90aWZmOyBjbG91ZC1vcHRpbWl6ZWQ9dHJ1ZSIsInRpdGxlIjoiUFNTY2VuZTRCYW5kIEFuYWx5dGljIEdlb1RJRkYiLCJwbDp0eXBlIjoiaHR0cHM6Ly9hcGkucGxhbmV0LmNvbS9kYXRhL3YxL2Fzc2V0LXR5cGVzL2FuYWx5dGljIiwicm9sZXMiOlsiZGF0YSJdfSwiYW5hbHl0aWNfeG1sIjp7ImhyZWYiOiJodHRwczovL3N0b3JhZ2UuZ29vZ2xlYXBpcy5jb20vcGRkLXN0YWMvZGlzYXN0ZXJzL2h1cnJpY2FuZS1oYXJ2ZXkvMDgzMS8yMDE3MDgzMV8xNzI3NTRfMTAxY18zQl9BbmFseXRpY01TX21ldGFkYXRhLnhtbCIsInR5cGUiOiJ0ZXh0L3htbCIsInRpdGxlIjoiUFNTY2VuZTRCYW5kIFhNTCBNZXRhZGF0YSIsInBsOnR5cGUiOiJodHRwczovL2FwaS5wbGFuZXQuY29tL2RhdGEvdjEvYXNzZXQtdHlwZXMvYW5hbHl0aWNfeG1sIiwicm9sZXMiOlsibWV0YWRhdGEiXX0sInVkbSI6eyJocmVmIjoiaHR0cHM6Ly9zdG9yYWdlLmdvb2dsZWFwaXMuY29tL3BkZC1zdGFjL2Rpc2FzdGVycy9odXJyaWNhbmUtaGFydmV5LzA4MzEvMjAxNzA4MzFfMTcyNzU0XzEwMWNfM0JfQW5hbHl0aWNNU19ETl91ZG0udGlmIiwidHlwZSI6ImltYWdlL3ZuZC5zdGFjLmdlb3RpZmY7IGNsb3VkLW9wdGltaXplZD10cnVlIiwidGl0bGUiOiJQU1NjZW5lNEJhbmQgVW51c2FibGUgRGF0YSBNYXNrIiwicGw6dHlwZSI6Imh0dHBzOi8vYXBpLnBsYW5ldC5jb20vZGF0YS92MS9hc3NldC10eXBlcy91ZG0iLCJyb2xlcyI6WyJ1ZG0iXX0sInZpc3VhbCI6eyJocmVmIjoiaHR0cHM6Ly9zdG9yYWdlLmdvb2dsZWFwaXMuY29tL3BkZC1zdGFjL2Rpc2FzdGVycy9odXJyaWNhbmUtaGFydmV5LzA4MzEvMjAxNzA4MzFfMTcyNzU0XzEwMWNfM2JfVmlzdWFsLnRpZiIsInR5cGUiOiJpbWFnZS92bmQuc3RhYy5nZW90aWZmOyBjbG91ZC1vcHRpbWl6ZWQ9dHJ1ZSIsInRpdGxlIjoiUFNTY2VuZTNCYW5kIFZpc3VhbCBHZW9USUZGIiwicGw6dHlwZSI6Imh0dHBzOi8vYXBpLnBsYW5ldC5jb20vZGF0YS92MS9hc3NldC10eXBlcy92aXN1YWwiLCJyb2xlcyI6WyJ2aXN1YWwiXX19LCJiYm94IjpbLTk2LjA0MDAxNjIzODA5OTEzLDI5LjUxMjM0NDM1OTU3NjEwNSwtOTUuNzgyMTM4MjY1Mjg5MywyOS42MjMzMDgxNzA2MDUxOF0sInN0YWNfZXh0ZW5zaW9ucyI6WyJlbyIsInZpZXciLCJwcm9qIl0sImNvbGxlY3Rpb24iOiJwbGFuZXQtZGlzYXN0ZXItZGF0YSJ9

which decodes to 

{"type":"Feature","stac_version":"1.0.0-beta.2","id":"20170831_172754_101c","properties":{"datetime":"2017-08-31T17:27:54.960530Z","instruments":["PS2"],"platform":"101c","constellation":"planetscope","created":"2017-10-27T12:14:07Z","updated":"2017-12-07T07:37:28Z","gsd":2.6,"eo:cloud_cover":2,"view:sun_azimuth":145.4,"view:sun_elevation":65.1,"view:off_nadir":0.2,"proj:epsg":32615,"pl:anomalous_pixels":0.01,"pl:columns":8307,"pl:ground_control":true,"pl:item_type":"PSScene4Band","pl:origin_x":205503,"pl:origin_y":3280320,"pl:pixel_resolution":3,"pl:quality_category":"test","pl:rows":3918,"pl:strip_id":"724553","pl:usable_data":0},"geometry":{"type":"Polygon","coordinates":[[[-96.03352746458962,29.549057939888584],[-96.03810006842333,29.569454227086553],[-96.03730797076582,29.569591543553933],[-96.04001623809913,29.581290775117473],[-95.79781720037577,29.62330817060518],[-95.78672161779234,29.575096137087073],[-95.7821382652893,29.55447424651818],[-96.0250376017418,29.512344359576105],[-96.03352746458962,29.549057939888584]]]},"links":[{"rel":"collection","href":"../../../collection.json"},{"rel":"root","href":"../../../collection.json","type":"application/json"},{"rel":"parent","href":"../catalog.json","type":"application/json"}],"assets":{"thumbnail":{"href":"https://storage.googleapis.com/pdd-stac/disasters/hurricane-harvey/0831/20170831_172754_101c_thumb_large.png","type":"image/png","title":"Thumbnail","roles":["thumbnail"]},"analytic":{"href":"https://storage.googleapis.com/pdd-stac/disasters/hurricane-harvey/0831/20170831_172754_101c_3B_AnalyticMS.tif","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"PSScene4Band Analytic GeoTIFF","pl:type":"https://api.planet.com/data/v1/asset-types/analytic","roles":["data"]},"analytic_xml":{"href":"https://storage.googleapis.com/pdd-stac/disasters/hurricane-harvey/0831/20170831_172754_101c_3B_AnalyticMS_metadata.xml","type":"text/xml","title":"PSScene4Band XML Metadata","pl:type":"https://api.planet.com/data/v1/asset-types/analytic_xml","roles":["metadata"]},"udm":{"href":"https://storage.googleapis.com/pdd-stac/disasters/hurricane-harvey/0831/20170831_172754_101c_3B_AnalyticMS_DN_udm.tif","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"PSScene4Band Unusable Data Mask","pl:type":"https://api.planet.com/data/v1/asset-types/udm","roles":["udm"]},"visual":{"href":"https://storage.googleapis.com/pdd-stac/disasters/hurricane-harvey/0831/20170831_172754_101c_3b_Visual.tif","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"PSScene3Band Visual GeoTIFF","pl:type":"https://api.planet.com/data/v1/asset-types/visual","roles":["visual"]}},"bbox":[-96.04001623809913,29.512344359576105,-95.7821382652893,29.62330817060518],"stac_extensions":["eo","view","proj"],"collection":"planet-disaster-data"}
vincentsarago commented 3 years ago

IMO this is quite a custom feature which could easily be fixed by a custom STACReader on your side (and then used in titiler)

import attr
import pystac
from rio_tiler.io import STACReader

@attr.s
class MyCustomSTACReader(STACReader)

def __attrs_post_init__(self):
    if self.filepath and self.filepath.startswith("data:"):
        self.items = pystac.Item(json.loads(base64decode(self.filepath.split("data:")[1]))
        self.filepath = None
    super(). __attrs_post_init__()
vincentsarago commented 3 years ago

@DanielJDufour Are we good here?

FYI, I've played a bit with the idea and here is what I'm using 

import json
from base64 import b64decode

import attr
import pystac
from rio_tiler.io import STACReader

@attr.s
class MyCustomSTACReader(STACReader):
    """Custom STACReader."""

    def __attrs_post_init__(self):
        """Decode data: urls."""
        if self.filepath and self.filepath.startswith("stac://"):
            self.item = pystac.Item.from_dict(
                json.loads(
                    b64decode(
                        self.filepath.replace("stac://", "")
                    )
                )
            )
            self.filepath = None

        super(). __attrs_post_init__()

I added this in a titiler demo. You'll note that I'm using stac:// instead of data:application/json;base64, because I had weird URL parking issue.

vincentsarago commented 3 years ago

closing for now