developmentseed / titiler

Build your own Raster dynamic map tile services
https://developmentseed.org/titiler/
MIT License
792 stars 165 forks source link

Intermittent "Out of range float values are not JSON compliant" errors #413

Closed Plantain closed 2 years ago

Plantain commented 2 years ago

It seems like Titiler can get itself in a state where subsequent reads to the same file return JSON errors. I am not sure if this is related to #409 yet.

It seems fairly easy to reproduce, I can reproduce it locally or on titiler.xyz just by repeatedly hitting tilejson.json, after about 2-3 hits it starts returning errors.

If I kill the worker locally it works fine for a few then starts erroring, returning 500's with: {"detail":"Out of range float values are not JSON compliant"}

For example:

while true; do curl 'http://localhost:8000/cog/tilejson.json?url=http://static2.skysight.io/demo.tiff&bidx=1&bidx=2&bidx=3'; done {"tilejson":"2.2.0","version":"1.0.0","scheme":"xyz","tiles":["http://localhost:8000/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=http%3A%2F%2Fstatic2.skysight.io%2Fdemo.tiff&bidx=1&bidx=2&bidx=3"],"minzoom":1,"maxzoom":7,"bounds":[-180.0,-90.0,180.0,90.0],"center":[0.0,0.0,1]}{"tilejson":"2.2.0","version":"1.0.0","scheme":"xyz","tiles":["http://localhost:8000/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=http%3A%2F%2Fstatic2.skysight.io%2Fdemo.tiff&bidx=1&bidx=2&bidx=3"],"minzoom":1,"maxzoom":7,"bounds":[-180.0,-90.0,180.0,90.0],"center":[0.0,0.0,1]}{"tilejson":"2.2.0","version":"1.0.0","scheme":"xyz","tiles":["http://localhost:8000/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=http%3A%2F%2Fstatic2.skysight.io%2Fdemo.tiff&bidx=1&bidx=2&bidx=3"],"minzoom":1,"maxzoom":7,"bounds":[-180.0,-90.0,180.0,90.0],"center":[0.0,0.0,1]}{"tilejson":"2.2.0","version":"1.0.0","scheme":"xyz","tiles":["http://localhost:8000/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=http%3A%2F%2Fstatic2.skysight.io%2Fdemo.tiff&bidx=1&bidx=2&bidx=3"],"minzoom":1,"maxzoom":7,"bounds":[-180.0,-90.0,180.0,90.0],"center":[0.0,0.0,1]}{"tilejson":"2.2.0","version":"1.0.0","scheme":"xyz","tiles":["http://localhost:8000/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=http%3A%2F%2Fstatic2.skysight.io%2Fdemo.tiff&bidx=1&bidx=2&bidx=3"],"minzoom":1,"maxzoom":7,"bounds":[-180.0,-90.0,180.0,90.0],"center":[0.0,0.0,1]}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}

Same result on titiler.xyz: while true; do curl 'https://titiler.xyz/cog/tilejson.json?url=http://static2.skysight.io/demo.tiff&bidx=1&bidx=2&bidx=3'; done

Stacktrace from local server:


  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/titiler/core/middleware.py", line 56, in dispatch
    response = await call_next(request)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 45, in call_next
    task.result()
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 38, in coro
    await self.app(scope, receive, send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/titiler/core/middleware.py", line 92, in dispatch
    response = await call_next(request)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 45, in call_next
    task.result()
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 38, in coro
    await self.app(scope, receive, send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/titiler/core/middleware.py", line 38, in dispatch
    response = await call_next(request)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 45, in call_next
    task.result()
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 38, in coro
    await self.app(scope, receive, send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette_cramjam/middleware.py", line 77, in __call__
    await self.app(scope, receive, send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/middleware/cors.py", line 78, in __call__
    await self.app(scope, receive, send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/fastapi/routing.py", line 243, in app
    response = actual_response_class(response_data, **response_args)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/responses.py", line 53, in __init__
    self.body = self.render(content)
  File "/home/plantain/titiler/venv/lib/python3.9/site-packages/starlette/responses.py", line 162, in render
    return json.dumps(
  File "/usr/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
Plantain commented 2 years ago

I think I have tracked this all the way down to rio_tiler, in particular rio_tiler/io/base.py and geographic_bounds.

This method semi-randomly throws "Reprojection failed, err = -20, further errors will be suppressed on the transform object.", in which case it defaults to (-180.0, -90.0, 180.0, 90.0), and occasionally succeeds and returns (-179.9624834120246, -52.94556026404906, inf, inf). The inf breaks the json encoding.

    def geographic_bounds(self) -> BBox:
        """return bounds in WGS84."""
        try:
            bounds = transform_bounds(
                self.crs,
                self.geographic_crs,
                *self.bounds,
                densify_pts=21,
            )   
        except:  # noqa
            warnings.warn(
                "Cannot dertermine bounds in WGS84, will default to (-180.0, -90.0, 180.0, 90.0).",
                UserWarning,
            )
            bounds = (-180.0, -90, 180.0, 90)

        return bounds

It looks like this behaviour is known about because in tile_exists there is this illuminating comment:


        try:
            dataset_bounds = transform_bounds(
                self.crs,
                self.tms.rasterio_crs,
                *self.bounds,
                densify_pts=21,
            )
        except:  # noqa
            # HACK: gdal will first throw an error for invalid transformation
            # but if retried it will then pass.
            # Note: It might return `+/-inf` values
            dataset_bounds = transform_bounds(
                self.crs,
                self.tms.rasterio_crs,
                *self.bounds,
                densify_pts=21,
            )
vincentsarago commented 2 years ago

@Plantain thanks for reporting this.

Your file is interesting:

In [2]: with COGReader("http://static2.skysight.io/demo.tiff") as cog:
   ...:     print(cog.geographic_bounds)
   ...:     print(cog.geographic_bounds)
   ...: 
NoOverviewWarning: The dataset has no Overviews. rio-tiler performances might be impacted.
  warnings.warn(
UserWarning: Cannot dertermine bounds in WGS84, will default to (-180.0, -90.0, 180.0, 90.0).
  warnings.warn(
(-180.0, -90, 180.0, 90)
(-179.9624834120246, -52.94556026404906, 174.19870167468366, 52.945560264049064)

At first GDAL will raise an exception but success on consecutive tries. see: https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/io/base.py#L52-L69

Locally with GDAL 3.3.3, it works fine but for some reason the GDAL/Version (using rasterio wheels) deployed at titiler.xyz will returned invalid values. I would tend to think that this is either a Data issue or a GDAL issue that was fixed in recent GDAL version.

vincentsarago commented 2 years ago

sorry I posted my reply at the same time as yours :-)

This method semi-randomly throws "Reprojection failed, err = -20, further errors will be suppressed on the transform object.", in which case it defaults to (-180.0, -90.0, 180.0, 90.0), and occasionally succeeds and returns (-179.9624834120246, -52.94556026404906, inf, inf). The inf breaks the json encoding.

I think I can make a quick fix for this. I'll closes this issue here and make a PR in rio-tiler

Plantain commented 2 years ago

Ok great, thanks for looking into this so quickly. It's generated by GDAL with -of COG, so I will figure out what's going wrong with the .tiff.

vincentsarago commented 2 years ago

FYI:

$ rio cogeo info http://static2.skysight.io/demo.tiff
Driver: GTiff
File: http://static2.skysight.io/demo.tiff
COG: False
Compression: JPEG
ColorSpace: None

Profile
    Width:            17964
    Height:           17964
    Bands:            4
    Tiled:            False
    Dtype:            uint8
    NoData:           0.0
    Alpha Band:       True
    Internal Mask:    False
    Interleave:       PIXEL
    ColorMap:         False
    ColorInterp:      ('red', 'green', 'blue', 'alpha')
    Scales:           (1.0, 1.0, 1.0, 1.0)
    Offsets:          (0.0, 0.0, 0.0, 0.0)

Geo
    Crs:              PROJCS["unknown",GEOGCS["unknown",DATUM["unknown",SPHEROID["GRS 1980",6378137,298.257222096042]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433]],PROJECTION["Geostationary_Satellite"],PARAMETER["central_meridian",-137],PARAMETER["satellite_height",35786023],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],EXTENSION["PROJ4","+proj=geos +sweep=x +lon_0=-137 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs"]]
    Origin:           (-4500020.820204, 4500020.820204)
    Resolution:       (501.0043219999999, -501.0043219999999)
    BoundingBox:      (-4500020.820204, -4500020.820203998, 4500020.820203998, 4500020.820204)
    MinZoom:          1
    MaxZoom:          7

Image Metadata
    AREA_OR_POINT: Area
    TIFFTAG_DATETIME: 2021:12:03 05:30:32

Image Structure
    COMPRESSION: JPEG
    INTERLEAVE: PIXEL
    JPEGTABLESMODE: 1
    JPEG_QUALITY: 75

IFD
    Id      Size           BlockSize     Decimation           
    0       17964x17964    17964x16      0

COG Validation info
    - The file is greater than 512xH or 512xW, but is not tiled (error)
    - The file is greater than 512xH or 512xW, it is recommended to include internal overviews (warning)

Creating correct COG:

$ rio cogeo create http://static2.skysight.io/demo.tiff demo_cog.tif -p jpeg --add-mask --bidx 1,2,3 --use-cog-driver --overview-blocksize 512

$ rio cogeo info demo_cog.tif 
Driver: GTiff
File: /Users/vincentsarago/Dev/DevSeed/titiler/demo_cog.tif
COG: True
Compression: JPEG
ColorSpace: YCbCr

Profile
    Width:            17964
    Height:           17964
    Bands:            3
    Tiled:            True
    Dtype:            uint8
    NoData:           None
    Alpha Band:       False
    Internal Mask:    True
    Interleave:       PIXEL
    ColorMap:         False
    ColorInterp:      ('red', 'green', 'blue')
    Scales:           (1.0, 1.0, 1.0)
    Offsets:          (0.0, 0.0, 0.0)

Geo
    Crs:              PROJCS["unknown",GEOGCS["unknown",DATUM["unknown",SPHEROID["GRS 1980",6378137,298.257222096042]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433]],PROJECTION["Geostationary_Satellite"],PARAMETER["central_meridian",-137],PARAMETER["satellite_height",35786023],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],EXTENSION["PROJ4","+proj=geos +sweep=x +lon_0=-137 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs"]]
    Origin:           (-4500020.820204, 4500020.820204)
    Resolution:       (501.0043219999999, -501.0043219999999)
    BoundingBox:      (-4500020.820204, -4500020.820203998, 4500020.820203998, 4500020.820204)
    MinZoom:          1
    MaxZoom:          7

Image Metadata
    AREA_OR_POINT: Area
    OVR_RESAMPLING_ALG: NEAREST
    TIFFTAG_DATETIME: 2021:12:03 05:30:32

Image Structure
    COMPRESSION: YCbCr JPEG
    INTERLEAVE: PIXEL
    JPEGTABLESMODE: 1
    JPEG_QUALITY: 75
    LAYOUT: COG
    SOURCE_COLOR_SPACE: YCbCr

IFD
    Id      Size           BlockSize     Decimation           
    0       17964x17964    512x512       0
    1       8982x8982      512x512       2
    2       4491x4491      512x512       4
    3       2246x2246      512x512       8
    4       1123x1123      512x512       16
    5       562x562        512x512       32
    6       281x281        512x512       64
vincentsarago commented 2 years ago

fixed in https://github.com/cogeotiff/rio-tiler/pull/467, will publish a new version of rio-tiler and update titiler stack later today

Plantain commented 2 years ago

Regarding "it uses a CRS/bounds that cannot be transformed to WGS84" - is this necessarily a problem?

I uploaded an actual COG in the same projection to static2.skysight.io/demo2.tiff , which gdalwarp will warp quite happily to EPSG:3857, but titiler won't, failing with ({"detail":"tolerance condition error"}) - not sure if that is the same root cause or not.

vincentsarago commented 2 years ago

Regarding "it uses a CRS/bounds that cannot be transformed to WGS84" - is this necessarily a problem?

No it's not a real problem, at least for earth based dataset. The problem with GOES data (and other geostationary dataset) is that their Field of view is usually bigger than the earth, which is why we can't properly convert the dataset bounds to WGS84 (not WebMercator).

but titiler won't, failing with ({"detail":"tolerance condition error"}) - not sure if that is the same root cause or not.

To me it seems to work well 🤷, some tiles will fail (because we cannot get the bounds right, the tiler will try to read tiles for the whole earth, and some will fail.

Plantain commented 2 years ago

Ok - do see the tiles if I hit the tiles themselves, but it fails to load in the viewer for me because of the info request failing. curl 'https://titiler.xyz/cog/info?url=http://static2.skysight.io/demo2.tiff' {"detail":"tolerance condition error"} Is that this issue, or a separate issue?

vincentsarago commented 2 years ago

oh that's another issue but kinda related to the original one.

In [1]: from rio_tiler.io import COGReader

In [2]: with COGReader("http://static2.skysight.io/demo2.tiff") as cog:
   ...:     print(cog.info())
   ...: 
---------------------------------------------------------------------------
CPLE_AppDefinedError                      Traceback (most recent call last)
<ipython-input-2-585e24152792> in <module>
      1 with COGReader("http://static2.skysight.io/demo2.tiff") as cog:
----> 2     print(cog.info())
      3 

~/Dev/CogeoTiff/rio-tiler/rio_tiler/io/cogeo.py in info(self)
    208             return self.dataset.descriptions[ix - 1] or ""
    209 
--> 210         if has_alpha_band(self.dataset):
    211             nodata_type = "Alpha"
    212         elif has_mask_band(self.dataset):

~/Dev/CogeoTiff/rio-tiler/rio_tiler/utils.py in has_alpha_band(src_dst)
    309     if (
    310         any([MaskFlags.alpha in flags for flags in src_dst.mask_flag_enums])
--> 311         or ColorInterp.alpha in src_dst.colorinterp
    312     ):
    313         return True

~/Dev/vincentsarago/rasterio/rasterio/_base.pyx in rasterio._base.DatasetBase.colorinterp.__get__()

~/Dev/vincentsarago/rasterio/rasterio/_err.pyx in rasterio._err.exc_wrap_int()

CPLE_AppDefinedError: tolerance condition error

I'll open an issue in rio-tiler

TBH, when working with GOES dataset, we found that it was easier to first convert them to WGS84 COG and then use them in TiTiler like service.

Plantain commented 2 years ago

Ok, thanks for looking into this.

TBH, when working with GOES dataset, we found that it was easier to first convert them to WGS84 COG and then use them in TiTiler like service.

I did try that, but GDAL seems to resample it to a pixel size of 1200 using TILING_SCHEME=GoogleMapsCompatible (substantially less detail), plus it's slower to create especially when most imagery will never be looked at...

vincentsarago commented 2 years ago

@Plantain as mentioned in https://github.com/cogeotiff/rio-tiler/issues/468, this is purely a rasterio/gdal issue.