Closed Plantain closed 2 years ago
I think I have tracked this all the way down to rio_tiler, in particular rio_tiler/io/base.py and geographic_bounds.
This method semi-randomly throws "Reprojection failed, err = -20, further errors will be suppressed on the transform object.", in which case it defaults to (-180.0, -90.0, 180.0, 90.0), and occasionally succeeds and returns (-179.9624834120246, -52.94556026404906, inf, inf). The inf breaks the json encoding.
def geographic_bounds(self) -> BBox:
"""return bounds in WGS84."""
try:
bounds = transform_bounds(
self.crs,
self.geographic_crs,
*self.bounds,
densify_pts=21,
)
except: # noqa
warnings.warn(
"Cannot dertermine bounds in WGS84, will default to (-180.0, -90.0, 180.0, 90.0).",
UserWarning,
)
bounds = (-180.0, -90, 180.0, 90)
return bounds
It looks like this behaviour is known about because in tile_exists there is this illuminating comment:
try:
dataset_bounds = transform_bounds(
self.crs,
self.tms.rasterio_crs,
*self.bounds,
densify_pts=21,
)
except: # noqa
# HACK: gdal will first throw an error for invalid transformation
# but if retried it will then pass.
# Note: It might return `+/-inf` values
dataset_bounds = transform_bounds(
self.crs,
self.tms.rasterio_crs,
*self.bounds,
densify_pts=21,
)
@Plantain thanks for reporting this.
Your file is interesting:
In [2]: with COGReader("http://static2.skysight.io/demo.tiff") as cog:
...: print(cog.geographic_bounds)
...: print(cog.geographic_bounds)
...:
NoOverviewWarning: The dataset has no Overviews. rio-tiler performances might be impacted.
warnings.warn(
UserWarning: Cannot dertermine bounds in WGS84, will default to (-180.0, -90.0, 180.0, 90.0).
warnings.warn(
(-180.0, -90, 180.0, 90)
(-179.9624834120246, -52.94556026404906, 174.19870167468366, 52.945560264049064)
At first GDAL will raise an exception but success on consecutive tries. see: https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/io/base.py#L52-L69
Locally with GDAL 3.3.3, it works fine but for some reason the GDAL/Version (using rasterio wheels) deployed at titiler.xyz will returned invalid values. I would tend to think that this is either a Data issue or a GDAL issue that was fixed in recent GDAL version.
sorry I posted my reply at the same time as yours :-)
This method semi-randomly throws "Reprojection failed, err = -20, further errors will be suppressed on the transform object.", in which case it defaults to (-180.0, -90.0, 180.0, 90.0), and occasionally succeeds and returns (-179.9624834120246, -52.94556026404906, inf, inf). The inf breaks the json encoding.
I think I can make a quick fix for this. I'll closes this issue here and make a PR in rio-tiler
Ok great, thanks for looking into this so quickly. It's generated by GDAL with -of COG, so I will figure out what's going wrong with the .tiff.
FYI:
$ rio cogeo info http://static2.skysight.io/demo.tiff
Driver: GTiff
File: http://static2.skysight.io/demo.tiff
COG: False
Compression: JPEG
ColorSpace: None
Profile
Width: 17964
Height: 17964
Bands: 4
Tiled: False
Dtype: uint8
NoData: 0.0
Alpha Band: True
Internal Mask: False
Interleave: PIXEL
ColorMap: False
ColorInterp: ('red', 'green', 'blue', 'alpha')
Scales: (1.0, 1.0, 1.0, 1.0)
Offsets: (0.0, 0.0, 0.0, 0.0)
Geo
Crs: PROJCS["unknown",GEOGCS["unknown",DATUM["unknown",SPHEROID["GRS 1980",6378137,298.257222096042]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433]],PROJECTION["Geostationary_Satellite"],PARAMETER["central_meridian",-137],PARAMETER["satellite_height",35786023],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],EXTENSION["PROJ4","+proj=geos +sweep=x +lon_0=-137 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs"]]
Origin: (-4500020.820204, 4500020.820204)
Resolution: (501.0043219999999, -501.0043219999999)
BoundingBox: (-4500020.820204, -4500020.820203998, 4500020.820203998, 4500020.820204)
MinZoom: 1
MaxZoom: 7
Image Metadata
AREA_OR_POINT: Area
TIFFTAG_DATETIME: 2021:12:03 05:30:32
Image Structure
COMPRESSION: JPEG
INTERLEAVE: PIXEL
JPEGTABLESMODE: 1
JPEG_QUALITY: 75
IFD
Id Size BlockSize Decimation
0 17964x17964 17964x16 0
COG Validation info
- The file is greater than 512xH or 512xW, but is not tiled (error)
- The file is greater than 512xH or 512xW, it is recommended to include internal overviews (warning)
Creating correct COG:
$ rio cogeo create http://static2.skysight.io/demo.tiff demo_cog.tif -p jpeg --add-mask --bidx 1,2,3 --use-cog-driver --overview-blocksize 512
$ rio cogeo info demo_cog.tif
Driver: GTiff
File: /Users/vincentsarago/Dev/DevSeed/titiler/demo_cog.tif
COG: True
Compression: JPEG
ColorSpace: YCbCr
Profile
Width: 17964
Height: 17964
Bands: 3
Tiled: True
Dtype: uint8
NoData: None
Alpha Band: False
Internal Mask: True
Interleave: PIXEL
ColorMap: False
ColorInterp: ('red', 'green', 'blue')
Scales: (1.0, 1.0, 1.0)
Offsets: (0.0, 0.0, 0.0)
Geo
Crs: PROJCS["unknown",GEOGCS["unknown",DATUM["unknown",SPHEROID["GRS 1980",6378137,298.257222096042]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433]],PROJECTION["Geostationary_Satellite"],PARAMETER["central_meridian",-137],PARAMETER["satellite_height",35786023],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],EXTENSION["PROJ4","+proj=geos +sweep=x +lon_0=-137 +h=35786023 +x_0=0 +y_0=0 +ellps=GRS80 +units=m +no_defs"]]
Origin: (-4500020.820204, 4500020.820204)
Resolution: (501.0043219999999, -501.0043219999999)
BoundingBox: (-4500020.820204, -4500020.820203998, 4500020.820203998, 4500020.820204)
MinZoom: 1
MaxZoom: 7
Image Metadata
AREA_OR_POINT: Area
OVR_RESAMPLING_ALG: NEAREST
TIFFTAG_DATETIME: 2021:12:03 05:30:32
Image Structure
COMPRESSION: YCbCr JPEG
INTERLEAVE: PIXEL
JPEGTABLESMODE: 1
JPEG_QUALITY: 75
LAYOUT: COG
SOURCE_COLOR_SPACE: YCbCr
IFD
Id Size BlockSize Decimation
0 17964x17964 512x512 0
1 8982x8982 512x512 2
2 4491x4491 512x512 4
3 2246x2246 512x512 8
4 1123x1123 512x512 16
5 562x562 512x512 32
6 281x281 512x512 64
fixed in https://github.com/cogeotiff/rio-tiler/pull/467, will publish a new version of rio-tiler and update titiler stack later today
Regarding "it uses a CRS/bounds that cannot be transformed to WGS84" - is this necessarily a problem?
I uploaded an actual COG in the same projection to static2.skysight.io/demo2.tiff , which gdalwarp will warp quite happily to EPSG:3857, but titiler won't, failing with ({"detail":"tolerance condition error"}) - not sure if that is the same root cause or not.
Regarding "it uses a CRS/bounds that cannot be transformed to WGS84" - is this necessarily a problem?
No it's not a real problem, at least for earth based dataset. The problem with GOES data (and other geostationary dataset) is that their Field of view is usually bigger than the earth, which is why we can't properly convert the dataset bounds to WGS84 (not WebMercator).
but titiler won't, failing with ({"detail":"tolerance condition error"}) - not sure if that is the same root cause or not.
To me it seems to work well 🤷, some tiles will fail (because we cannot get the bounds right, the tiler will try to read tiles for the whole earth, and some will fail.
Ok - do see the tiles if I hit the tiles themselves, but it fails to load in the viewer for me because of the info request failing. curl 'https://titiler.xyz/cog/info?url=http://static2.skysight.io/demo2.tiff' {"detail":"tolerance condition error"} Is that this issue, or a separate issue?
oh that's another issue but kinda related to the original one.
In [1]: from rio_tiler.io import COGReader
In [2]: with COGReader("http://static2.skysight.io/demo2.tiff") as cog:
...: print(cog.info())
...:
---------------------------------------------------------------------------
CPLE_AppDefinedError Traceback (most recent call last)
<ipython-input-2-585e24152792> in <module>
1 with COGReader("http://static2.skysight.io/demo2.tiff") as cog:
----> 2 print(cog.info())
3
~/Dev/CogeoTiff/rio-tiler/rio_tiler/io/cogeo.py in info(self)
208 return self.dataset.descriptions[ix - 1] or ""
209
--> 210 if has_alpha_band(self.dataset):
211 nodata_type = "Alpha"
212 elif has_mask_band(self.dataset):
~/Dev/CogeoTiff/rio-tiler/rio_tiler/utils.py in has_alpha_band(src_dst)
309 if (
310 any([MaskFlags.alpha in flags for flags in src_dst.mask_flag_enums])
--> 311 or ColorInterp.alpha in src_dst.colorinterp
312 ):
313 return True
~/Dev/vincentsarago/rasterio/rasterio/_base.pyx in rasterio._base.DatasetBase.colorinterp.__get__()
~/Dev/vincentsarago/rasterio/rasterio/_err.pyx in rasterio._err.exc_wrap_int()
CPLE_AppDefinedError: tolerance condition error
I'll open an issue in rio-tiler
TBH, when working with GOES dataset, we found that it was easier to first convert them to WGS84 COG and then use them in TiTiler like service.
Ok, thanks for looking into this.
TBH, when working with GOES dataset, we found that it was easier to first convert them to WGS84 COG and then use them in TiTiler like service.
I did try that, but GDAL seems to resample it to a pixel size of 1200 using TILING_SCHEME=GoogleMapsCompatible (substantially less detail), plus it's slower to create especially when most imagery will never be looked at...
@Plantain as mentioned in https://github.com/cogeotiff/rio-tiler/issues/468, this is purely a rasterio/gdal issue.
It seems like Titiler can get itself in a state where subsequent reads to the same file return JSON errors. I am not sure if this is related to #409 yet.
It seems fairly easy to reproduce, I can reproduce it locally or on titiler.xyz just by repeatedly hitting tilejson.json, after about 2-3 hits it starts returning errors.
If I kill the worker locally it works fine for a few then starts erroring, returning 500's with: {"detail":"Out of range float values are not JSON compliant"}
For example:
while true; do curl 'http://localhost:8000/cog/tilejson.json?url=http://static2.skysight.io/demo.tiff&bidx=1&bidx=2&bidx=3'; done {"tilejson":"2.2.0","version":"1.0.0","scheme":"xyz","tiles":["http://localhost:8000/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=http%3A%2F%2Fstatic2.skysight.io%2Fdemo.tiff&bidx=1&bidx=2&bidx=3"],"minzoom":1,"maxzoom":7,"bounds":[-180.0,-90.0,180.0,90.0],"center":[0.0,0.0,1]}{"tilejson":"2.2.0","version":"1.0.0","scheme":"xyz","tiles":["http://localhost:8000/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=http%3A%2F%2Fstatic2.skysight.io%2Fdemo.tiff&bidx=1&bidx=2&bidx=3"],"minzoom":1,"maxzoom":7,"bounds":[-180.0,-90.0,180.0,90.0],"center":[0.0,0.0,1]}{"tilejson":"2.2.0","version":"1.0.0","scheme":"xyz","tiles":["http://localhost:8000/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=http%3A%2F%2Fstatic2.skysight.io%2Fdemo.tiff&bidx=1&bidx=2&bidx=3"],"minzoom":1,"maxzoom":7,"bounds":[-180.0,-90.0,180.0,90.0],"center":[0.0,0.0,1]}{"tilejson":"2.2.0","version":"1.0.0","scheme":"xyz","tiles":["http://localhost:8000/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=http%3A%2F%2Fstatic2.skysight.io%2Fdemo.tiff&bidx=1&bidx=2&bidx=3"],"minzoom":1,"maxzoom":7,"bounds":[-180.0,-90.0,180.0,90.0],"center":[0.0,0.0,1]}{"tilejson":"2.2.0","version":"1.0.0","scheme":"xyz","tiles":["http://localhost:8000/cog/tiles/WebMercatorQuad/{z}/{x}/{y}@1x?url=http%3A%2F%2Fstatic2.skysight.io%2Fdemo.tiff&bidx=1&bidx=2&bidx=3"],"minzoom":1,"maxzoom":7,"bounds":[-180.0,-90.0,180.0,90.0],"center":[0.0,0.0,1]}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}{"detail":"Out of range float values are not JSON compliant"}
Same result on titiler.xyz: while true; do curl 'https://titiler.xyz/cog/tilejson.json?url=http://static2.skysight.io/demo.tiff&bidx=1&bidx=2&bidx=3'; done
Stacktrace from local server: