MBTiles: cannot handle zoom_level > 22

DogmaF commented 4 years ago

Expected behavior and actual behavior.

Calling gdaladdo on an mbtiles file should automatically add the subsampling factors: 2, 4, 8, 16 etc. But instead it returns an error message:

ERROR 6: zoom_level > 22 not supported

Steps to reproduce the problem.

Use gdal_translate to convert a geo-referenced tiff to mbtiles format. I got the error using a large geotiff (16578 x 21503 px):

gdal_translate myfile_crs_3857.tif myfile.mbtiles -of MBTILES

Add overviews implicitly:

gdaladdo myfile.mbtiles

Operating system

OSX 10.15.7 64 bit

GDAL version and provenance

GDAL 3.0.2 installed using conda

INPUT FILE DETAILS

gdalinfo TLTR_crs_3857.tif
Driver: GTiff/GeoTIFF
Files: TLTR_crs_3857.tif
Size is 16578, 21503
Coordinate System is:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        DATUM["World Geodetic System 1984",
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["Popular Visualisation Pseudo-Mercator",
        METHOD["Popular Visualisation Pseudo Mercator",
            ID["EPSG",1024]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting (X)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing (Y)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["unknown"],
        AREA["World - 85°S to 85°N"],
        BBOX[-85.06,-180,85.06,180]],
    ID["EPSG",3857]]
Data axis to CRS axis mapping: 1,2
Origin = (-10071418.949580254033208,3656755.453475835733116)
Pixel Size = (0.015054137124516,-0.015054137124516)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_DATETIME=2020:10:05 19:50:55
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  TIFFTAG_SOFTWARE=Adobe Photoshop 21.2 (Macintosh)
  TIFFTAG_XRESOLUTION=72
  TIFFTAG_YRESOLUTION=72
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-10071418.950, 3656755.453) ( 90d28'23.14"W, 31d11' 4.81"N)
Lower Left  (-10071418.950, 3656431.744) ( 90d28'23.14"W, 31d10'55.86"N)
Upper Right (-10071169.382, 3656755.453) ( 90d28'15.07"W, 31d11' 4.81"N)
Lower Right (-10071169.382, 3656431.744) ( 90d28'15.07"W, 31d10'55.86"N)
Center      (-10071294.166, 3656593.599) ( 90d28'19.11"W, 31d11' 0.34"N)
Band 1 Block=16578x1 Type=Byte, ColorInterp=Red
  Mask Flags: PER_DATASET ALPHA 
Band 2 Block=16578x1 Type=Byte, ColorInterp=Green
  Mask Flags: PER_DATASET ALPHA 
Band 3 Block=16578x1 Type=Byte, ColorInterp=Blue
  Mask Flags: PER_DATASET ALPHA 
Band 4 Block=16578x1 Type=Byte, ColorInterp=Alpha

jratike80 commented 4 years ago

Confirmed with GDAL 3.2.0 that has also a handy new utility "gdal_create". Your issue can be re-produced with these three commands

gdal_create -outsize 16578 21503 -a_srs epsg:3857 -co tiled=yes -co compress=deflate -a_ullr -10071418.950 3656755.453 -10071169.382 3656431.744 mbtest.tif

gdal_translate mbtest.tif mbtest.mbtiles -of MBTILES

gdaladdo mbtest.mbtiles
ERROR 6: zoom_level > 22 not supported

Reference: earlier discussion on the mailing list https://lists.osgeo.org/pipermail/gdal-dev/2020-October/052835.html.

DogmaF commented 4 years ago

Hi Jukka thanks for letting me know.

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, October 30th, 2020 at 1:08 AM, Jukka Rahkonen notifications@github.com wrote:

Confirmed with GDAL 3.2.0 that has also a handy new utility "gdal_create". Your issue can be re-produced with these three commands

gdal_create -outsize 16578 21503 -a_srs epsg:3857 -co tiled=yes -co compress=deflate -a_ullr -10071418.950 3656755.453 -10071169.382 3656431.744 mbtest.tif

gdal_translate mbtest.tif mbtest.mbtiles -of MBTILES

gdaladdo mbtest.mbtiles ERROR 6: zoom_level > 22 not supported

Reference: earlier discussion on the mailing list https://lists.osgeo.org/pipermail/gdal-dev/2020-October/052835.html.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

EvertEt commented 3 years ago

I am also running into the same error, and warping to 3857 as mentioned in the linked mailing thread does not change the behaviour (which makes sense since the sequence above with gdal_create also shows this behaviour).

Extracting the generated mbtiles with mb-util (https://pypi.org/project/mbutil/) shows indeed the included level is 23. The following metadata.json is also extracted.

{
    "name": "output",
    "type": "overlay",
    "description": "output",
    "version": "1.1",
    "format": "png",
    "bounds": "12.xx,51.xx,12.xx,51.xx",
    "minzoom": "23",
    "maxzoom": "23"
}

I have also tried the following with no change in behaviour as well gdal_translate output_3857.tif output.mbtiles -of MBTILES -co ZOOM_LEVEL_STRATEGY=LOWER

Do we know what the preferred solution is here? Allowing zoom level 23 based on extent to avoid the overflows, not allowing the generation of zoom level 23?

Let me know if I can help to get this resolved.

rouault commented 3 years ago

Allowing zoom level 23 based on extent to avoid the overflows

That would certainly be the prefered solution

EvertEt commented 3 years ago

In my current case, I only want to generate up to 22 if possible so it's a nice coincidence. I believe my case would benefit from a max zoom setting like gdal2tiles, and this would also provide a workaround for this I believe if we set it to 22.

EvertEt commented 3 years ago

I am not very familiar with C++ but looking at the source code I have two remarks I'd like to check. I am checking this part since it seems gdal_translate will use CreateCopy (https://github.com/OSGeo/gdal/blob/master/gdal/apps/gdal_translate_lib.cpp#L2141 correct me if I'm wrong) https://github.com/OSGeo/gdal/blob/master/gdal/frmts/mbtiles/mbtilesdataset.cpp#L3195

I have the feeling the zoom level loop should only go to 22, to avoid outputting MBTiles with a zoom level > 22 which will then not be able to be read by gdal. Instead of hardcoding the 22, I think it would be nice to have this changeable as per my last message.
Following on the mailing list https://lists.osgeo.org/pipermail/gdal-dev/2020-December/053131.html Might it be then that the zoom level is really 25 for my input file thus triggering the error case (similar to not finding a zoom level). Here a settable maximum would be helpful again if I am understanding it correctly.

I have the feeling my 2 problems on my 2 files (this issue and the one on the mailing list) are thus related to the same. Would you prefer this conversation to continue on one or the other (cc @jratike80 )

Thanks in advance

arredond commented 1 year ago

Hi! I just ran into this problem, same scenario as @DogmaT2 : Created an MBTiles from a GeoTIFF using gdal_translate, which creates level 24 tiles. When running gdaladdo, I get the ERROR 6: zoom_level > 22 not supported message.

From what I understand, the zoom level 22 limitation is hardcoded in the driver at read time, which is why it can be written (with gdal_translate) but not read (with gdaladdo). I'm guessing this limitation comes from the vector side of things (since the docs state a max zoom level of 22 for vector MBTiles creation, and most MVT writers are limited to a max zoom of 22).

@rouault would this be a matter of simply removing that restriction for raster MBTiles? And is there a way to differentiate a raster MBTiles from a vector one when reading? From a user perspective, the main difference is I'm using gdal* functions, like gdalinfo or gdal_translate instead of vector ones like ogr2ogr, but not sure how that translates to GDAL internals.

Also, for anyone encountering this same issue, I've found two workarounds (both suboptimal):

Use gdal2tiles.py to generate a directory of raster tiles (already with overlays), then use Mapbox's mbutil to collect them back into an MBTile.
Use gdal_translate to convert to a COG resampled to a specific zoom level (-co TILING_SCHEME=GoogleMapsCompatible -co ZOOM_LEVEL=22), then translate again into an MBTiles and add overviews with gdaladdo

jratike80 commented 1 year ago

Pixel size at zoom level is 3.7 cm and at z 24 about 9 millimeters at the equator. Of course images with such pixel size exists and I guess that you are working with such ultra-high resolution imagery. Am I right?

arredond commented 1 year ago

That's correct, this is very high-resolution crop imagery from UAVs (drones).

More in general, it feels wrong to have GDAL write an MBTiles that it can't read (though I understand where it's coming from). In this specific case, I'd be fine with gdal_translate just generating zoom level 22 tiles, instead of level 24 tiles, and having gdaladdo work with them

quietlychris commented 6 months ago

Just tagging in that I have also run into this issue while working with high-resolution UAV data; manually downscaling the GeoTIFF that the orthomosaic is generated from is a little clunky/error-prone (I sometimes wonder if I'm leaving resolution on the table in my files .mbtiles file), so it would be ideal to have an option to automatically generate Zoom 22 tiles instead.

OSGeo / gdal