Content for Elevation Tiles

jyutzler commented 9 years ago

We have agreed in principle to produce an extension for GeoPackage that stores gridded elevation data in the same tiling scheme previously specified for raster data. The question is what the content of those tiles should be.

One proposal is to use Binary Interleaved by Line 16-bit (BIL-16). While there is no formal specification for this format, it is commonly used and the GDAL implementation is considered the de facto standard.

An alternative is to use floating point values in a TIFF file. The downside to both of these alternatives is that they are not compressed formats. We would prefer to have a compressed option if possible.

pepijnve commented 9 years ago

TIFF does allow deflate compression of float data. Combined with the floating point horizontal differencing predictor you can get reasonable compression. A benefit of TIFF over BIL is that it's self describing. This obviously leads to duplicated metadata per tile, but makes processing of tile data by clients simpler. BIL needs external sidecar files (HDR iirc) for this.

mikereynolds commented 9 years ago

Another alternative we've employed to good effect is the storage of the grid data as a blob within a standard feature table, with the geom field being the bounding extents of the grid file/ blob data.

The blob is simply a byte[] of integer values taken from multiplying the original grid point values by a factor. The factor is stored against the table meta-data along with the number of rows and columns in the grid.

When a single grid value is required for a geo-location it can easily be extracted by querying the geom field (to get the correct record) and once known we get the correct offset in to the blob's byte[] and read the (height) value.

The trouble we found with compressing the data is that when 100's of calls (heights) are required across a large geographic area, many tiles need to be decompressed to get the point values. Obviously if the whole tile is required, the whole blob can be read and parsed in to a tile.

Using the above method, we package a 50m DTM for the UK in to one GeoPackage of 450mb. Running standard Compact on the gpkg gets it down by around 100mb (have to confirm tomorrow)

Mike

On 31 March 2015 at 18:51, Pepijn Van Eeckhoudt notifications@github.com wrote:

TIFF does allow deflate compression of float data. Combined with the floating point horizontal differencing predictor you can get reasonable compression. A benefit of TIFF over BIL is that it's self describing. This obviously leads to duplicated metadata per tile, but makes processing of tile data by clients simpler. BIL needs external sidecar files (HDR iirc) for this.

— Reply to this email directly or view it on GitHub https://github.com/opengeospatial/geopackage/issues/85#issuecomment-88187056 .

Mike Reynolds Managing Director, Augmented Technologies Ltd M: +44(0) 7703 727260 Email mike.reynolds@aug-tech.co.uk | Web http://www.augmented-technologies.co.uk | Twitter https://twitter.com/augmentedtech | LinkedIn http://uk.linkedin.com/pub/mike-reynolds/34/902/103 | Google+ https://plus.google.com/111810951008363072643 This email may contain confidential information or copyright material and is intended for the use of the addressee only. If you have received this email in error, please notify the sender by using the reply facility in your email software, and then delete the email.

jyutzler commented 9 years ago

@pepijnve I'd like to hear more on the metadata front. What metadata would you propose to put in the TIFF files that would be useful and not redundant to the metadata within the GeoPackage itself?

pepijnve commented 9 years ago

Image structure metadata. Width, height, scalene stride, sample format, bit depth, color space, compression method, etc.

When decoding a tile blob you need to know this information somehow. You can either put this in the blob itself making it easy to decode (all you need is the blob; in Java for instance you can simply do ImageIO#read and be done with it). Alternatively this data can be put in SQL tables, but then the decoding of an individual tile becomes more complicated.

At the moment RGB(A) is handled using the first approach. The tiles are complete PNG or JFIF streams. You don't need any information from the gpkg metadata tables in order to decode the image data of a tile. They also both have a magic number at the start of the file making it trivial to recognise which format it is based on the blob alone. Using TIFF would enable the same way of working for non RGB data.

A BIL file on the other hand has none of these niceties. It's basically a raw dump of binary values. It doesn't actually provide anything that TIFF can't do as well. Note that there's no reason to require TIFF to contain floating point data. You can put uncompressed, (un)signed 16-bit integer values in a TIFF as well. The overhead of the TIFF header in that case is very small.

pepijnve commented 9 years ago

@jyutzler btw, what I meant with duplicated metadata per tile was the image width/height, sample format, etc. Most likely this is going to be the same for all the tiles in a tile matrix. That's also already the case for the current PNG and JPEG blobs.

pepijnve commented 9 years ago

Another btw, BIL is an ESRI format. Best description I could find is at http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=BIL,_BIP,_and_BSQ_raster_files The section title 'The header file' describes the external .hdr file that you need in order to decode a .bil file correctly. This is the kind of information that would need to be stored somewhere in the geopackage tables if the tile blob is a bil file. An uncompressed TIFF file is essentially the same as a binary encoded version of a .hdr file and a .bil file concatenated together (gross simplificiation of course, but you get the idea).

jyutzler commented 9 years ago

@pepijnve I don't think an image format adds any value here. It isn't like anyone is going to display elevation data as an image directly. It is just overhead for overhead's sake and one more thing for the developers to worry about.

All that is needed is some form of byte array with a conventional byte order and that is what BIL is. (Calling BIL an Esri format is silly - it dates back to at least the 90s before Esri got involved in that kind of stuff.) A header file is unnecessary because everything we will want to know about a tile will either be in the specification (the one specific way of encoding this information) or the existing GeoPackage tables.

The only other thing that is left is compression which can just as easily be handled with an inflate/deflate option on the byte array itself.

jyutzler commented 9 years ago

@mikereynolds We acknowledge the cost of frequent random access of compressed byte arrays. Implementers will have to decide whether compression is worth that cost. My colleagues tell me that they generally don't perform analysis operations directly on the data at-rest but instead extract an area of interest and deal with it separately in memory. On Tuesday I will solicit more feedback on this subject.

pepijnve commented 9 years ago

@jyutzler well it's up to you guys now, but I don't agree. The value is not in using an image format, it's in using an existing container format that makes the tile blob self-contained (I.e. A self describing rectangle of sample values that can be decoded on its own). The overhead wrt the data is minimal and comparable to the rgb tiles and the convenience it buys you is worth it imo.

The esri format statement wasn't meant to be political or anything btw. That's simply the place where you can find the most information about it and I was under the impression that they defined this format.

KRyden commented 9 years ago

BIL, BIP, and BSQ imagery formats have been around for years - there is some variability in the implementations, so the ArcGIS documentation simply makes explicit what Esri implements for these formats - this was based on the most common implementations found.

Regarding the tile blob format - I agree that the tile blob should be self contained and fully self describing.

kjbacke commented 9 years ago

Classification: UNCLASSIFIED Caveats: NONE

Is the Gpkg telecom going on now? I dial-in but am the only one on the telecom.

-----Original Message----- From: Keith Ryden [mailto:notifications@github.com] Sent: Tuesday, April 14, 2015 10:54 AM To: opengeospatial/geopackage Subject: [EXTERNAL] Re: [geopackage] Content for Elevation Tiles (#85)

BIL, BIP, and BSQ imagery formats have been around for years - there is some variability in the implementations, so the ArcGIS documentation simply makes explicit what Esri implements for these formats - this was based on the most common implementations found.

Regarding the tile blob format - I agree that the tile blob should be self contained and fully self describing.

— Reply to this email directly or view it on GitHub https://github.com/opengeospatial/geopackage/issues/85#issuecomment-92889900 . https://github.com/notifications/beacon/AECrmhM_UycsjQc61vI71_Go-sQ3Txsnks5n_SFugaJpZM4D36Vv.gif

Classification: UNCLASSIFIED Caveats: NONE

rouault commented 9 years ago

FWIW, TIFF seems a good approach to me. It offers, among many things, int16 or floating point capabilities + optional compression + the same philosophy as the PNG or JPEG tiles, i.e. self describing

If you do a TIFF file of 256x256 pixels with a single strip, the overhead of the TIFF header is 146 bytes. See :

$ tiffdump tile.tif
tile.tif:
Magic: 0x4949 <little-endian> Version: 0x2a
Directory 0: offset 8 (0x8) next 0 (0)
ImageWidth (256) SHORT (3) 1<256>
ImageLength (257) SHORT (3) 1<256>
BitsPerSample (258) SHORT (3) 1<16>
Compression (259) SHORT (3) 1<1>
Photometric (262) SHORT (3) 1<1>
StripOffsets (273) LONG (4) 1<146>
SamplesPerPixel (277) SHORT (3) 1<1>
RowsPerStrip (278) SHORT (3) 1<256>
StripByteCounts (279) LONG (4) 1<131072>
PlanarConfig (284) SHORT (3) 1<1>
SampleFormat (339) SHORT (3) 1<2>

One drawback of TIFF is that libtiff has a record of security vulnerabilities due to the infinite number of variations of the format (esoterics codecs, etc...). But you could potentially put restrictions on the accepted variants of TIFF tiles. For example,

classic TIFF only (no BigTIFF)
byte ordering: little endian
one IFD only
IFD location : at offset 8
bit depth: UInt8, Int16 or Float32.
compression schemes : None or Deflate (with Predictor options).
strip organization with a single strip.
single channel, no extra sample. Actually, that's questionable. You might want an extra sample for unassociated alpha. Might be usefull for tiles at the border of the AOI. That could also be used for nodata, but that's a bit a heavy price to pay for nodata, since the bit depth of the alpha channel is the same as the main channel, whereas you'd need just a 1-bit channel. GDAL has invented its own TIFF tag to define the nodata value (http://www.awaresystems.be/imaging/tiff/tifftags/gdal_nodata.html). You could also have that an auxiliary IFD to define a transparency mask (NewSubfileType = 4 + PhotometricInterpretation = 4), but that belongs to the esoteric formulations of TIFF.

With such restrictions (excluding the complications with nodata/transparency), in the uncompressed case, it is essentially an header followed by a raw array of values, so you might not even need libtiff to decode it.

Regarding issues with frequent decompression of tiles, how different is it from current PNG and JPEG tiles ? This is something that can be handled by caching on the software reading the GPKG. GDAL for example has its block cache mechanism that avoid re-reading the file when querying tiles that were recently accessed.

A good thing would be to be able to know from the GPKG metadata tables what kind of data there is in a tile table : RGB/RGBA or elevation (perhaps addressed in other discussions) You would also need a way of knowing which unit (feet vs meter) the elevation values are. Perhaps through the WKT of the SRS ? You could also want to transport multispectral imagery some day too, so some of the above restrictions on TIFF might not be appropriate (namely single channel, and you would have to decide if both PlanarConfig=Contig or Separate are allowed)

royrathbun commented 9 years ago

We should get an opinion from the NITFS Technical Board, the technical focal point for imagery, and imagery-related standardization activities, within the GEOINT Standards Working Group. I can send them, Jim Durham and Steve Kerr, an invite to participate in the next telecon.

pepijnve commented 9 years ago

Another plus for TIFF is that WCS requires this as a base format IIRC. Or at least it's something that many WCS servers can generate as output format. Using TIFF as tile format would allow you to dump straight from a WCS service to GeoPackage without having to transcode the data. There's a bunch of caveats to this of course, but it would be very convenient. This would allow a similar simple extraction process for elevation data as is currently possible for RGB(A) imagery from a WMTS server. See #90 for additional comments on this.

jyutzler commented 9 years ago

No objection to unanimous consent for TIFF files.

opengeospatial / geopackage

Content for Elevation Tiles #85