Tiles are not being generated

[cgiovando]

I just uploaded this dataset:

https://map.openaerialmap.org/#/-72.39219640986414,18.52850169172233,16/0322112030020/59757e9946e19904aab0a90f?_k=nh9qpt

But I can't get tiles - also WMTS not working:

https://tiles.openaerialmap.org/59757c65a260c700113db212/0/1fb6708b-1c3a-4a7b-9ee7-a8385f3d1565/preview

It looks like previous datasets still work, so it's either something broken with latest commits, or an issue with this specific image?

[tombh]

Interesting. The underlying .tif is there: http://oin-hotosm.s3.amazonaws.com/59757c65a260c700113db212/0/1fb6708b-1c3a-4a7b-9ee7-a8385f3d1565.tif (it's 338MB so might not wanna click on that directly). I'd want to see the dynamic tiler's logs when the TMS is accessed to explore further. So I'll leave this one to @mojodna.

Also for reference, here is the TMS preview: https://tiles.openaerialmap.org/59757c65a260c700113db212/0/1fb6708b-1c3a-4a7b-9ee7-a8385f3d1565/preview#14/18.5285/-72.3922

And here is a sample tile, it returns HTTP 200, but is simply transparent: https://tiles.openaerialmap.org/59757c65a260c700113db212/0/1fb6708b-1c3a-4a7b-9ee7-a8385f3d1565/14/4896/7333.png

[smit1678]

It looks like previous datasets still work

@cgiovando Where are the previous datasets you're mentioning? I only see one ImageCat/RIT image in area near Port-au-Prince.

Posting output of gdalinfo on the originally uploaded file:

Driver: GTiff/GeoTIFF
Files: /vsicurl/https://s3.amazonaws.com/oam-uploader-production-temp/haiti_odm_example-6c29g5u8.tif
Size is 15142, 9876
Coordinate System is:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"],
    AUTHORITY["EPSG","3857"]]
Origin = (-8059680.024591206572950,2100166.197311825584620)
Pixel Size = (0.134405341233579,-0.134405341232950)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-8059680.025, 2100166.197) ( 72d24' 4.82"W, 18d32' 2.96"N)
Lower Left  (-8059680.025, 2098838.810) ( 72d24' 4.82"W, 18d31'22.25"N)
Upper Right (-8057644.859, 2100166.197) ( 72d22'59.00"W, 18d32' 2.96"N)
Lower Right (-8057644.859, 2098838.810) ( 72d22'59.00"W, 18d31'22.25"N)
Center      (-8058662.442, 2099502.504) ( 72d23'31.91"W, 18d31'42.61"N)
Band 1 Block=15142x1 Type=Int16, ColorInterp=Gray
  NoData Value=-32768
Band 2 Block=15142x1 Type=Int16, ColorInterp=Undefined
  NoData Value=-32768
Band 3 Block=15142x1 Type=Int16, ColorInterp=Undefined
  NoData Value=-32768
Band 4 Block=15142x1 Type=Int16, ColorInterp=Undefined
  NoData Value=-32768

[cgiovando]

@cgiovando Where are the previous datasets you're mentioning? I only see one ImageCat/RIT image in area near Port-au-Prince.

Sorry, I meant any other file already existing in OAM, not related to this project.

Adding @smathermather as this dataset came from him

[mojodna]

I suspect that it has to do with it being a 4-band Int16 file (3-band Int16 sources from DG are ok, fwiw), specifically how the mask is handled. (QGIS handles this fine; use the VRT, TIFF, and mask: http://oin-hotosm.s3.amazonaws.com/59757c65a260c700113db212/0/1fb6708b-1c3a-4a7b-9ee7-a8385f3d1565.tif http://oin-hotosm.s3.amazonaws.com/59757c65a260c700113db212/0/1fb6708b-1c3a-4a7b-9ee7-a8385f3d1565.tif.msk http://oin-hotosm.s3.amazonaws.com/59757c65a260c700113db212/0/1fb6708b-1c3a-4a7b-9ee7-a8385f3d1565.vrt)

The mask is a TIF containing a single-band of Byte data (0-255). It gets concatenated with the image data in https://github.com/hotosm/oam-dynamic-tiler/blob/master/tiler.py#L72, which converts it to Int16 (in this case) without scaling the values first (similar to what happens in https://github.com/hotosm/oam-dynamic-tiler/blob/master/tiler.py#L210). This is treated as RGBA, so when the alpha values get scaled, it's effectively transparent. (This is an edge case, but should be fixed.)

A secondary problem that seems like it would occur is that all of the color values are in the range 0..256, which will show up as black when rescaled from 16-bit to 8-bit. This seems to me more a problem with the source data than something we can fix in a generalized way.

(QGIS handles the second problem fine, potentially because it does contrast enhancement.)

@smathermather do you know how this ended up using Int16 as the data type? (The data is 8-bit.)

[mojodna]

(When turning off contrast enhancement in QGIS, the image gets a yellower tinge, but is still very much an aerial image, so I'm not sure how to interpret that, particularly relative to the values that come out of the DG imagery (which I'd want to double check, but I can't remember where those images were from).)

[smathermather]

Hmm, not sure why it's 16-bit, but it is, as surmised, stretched from 0-255. A simple gdal_translate writing it out as 8-bit with no stretch should be adequate for loading in OAM.

[mojodna]

@smathermather do you know how the image was produced in the first place? I'm wondering how common files like that are in the wild...

[smathermather]

I need to step back through the process to find out where it got converted to 16-bit. It was either in ODM (unlikely) or using the georeferencing tool in QGIS or using GDAL to trim the image.

[smathermather]

Now that I'm back to my laptop, I can verify that every copy of the data I have is BYTE, so I'm not sure where or when it became 16-bit. :) @cgiovando -- did you do any post-processing in gdal to prep it for OAM?