Open mojodna opened 8 years ago
I did a highly unscientific single test case, but I think it's fairly representative of the whole; take NED13 n32w090
, which is native Float32
data type:
.img
compressed in a .zip
file from USGS..img
Float32
with COMPRESS=DEFLATE, PREDICTOR=3
Int16
with COMPRESS=LZW, PREDICTOR=2
The reduction in file size by almost an order of magnitude is a benefit that has to be weighed against the improvement in precision allowed by Float32
. Just out of interest I tried:
Int32
data type to avoid clipping: 182MBOf course, the files will be larger, as we're retaining more of the information from the original. Sub-meter information can be very useful, for example in flat areas to avoid quantization artefacts appearing as false ridges, and many other uses. But its presence in the file comes at a cost for users who don't need that level of precision. It's not clear to me where the balance point lies.
Although it should be noted that not all of the data in the sub-meter range is information. For example, in tile 13/1326/3123
which, in the "terrarium" format, retains sub-meter information down to 1/256th of a meter in the blue band:
The "starburst" pattern is clearly an artefact, perhaps from merging multiple DEMs around a reference point. So although there's precision in NED, the accuracy should be carefully evaluated. LIDAR, likewise, has the capability of being incredibly precise, but care should be taken to avoid assumptions about its accuracy.
@nvkelso, what do you think about the precision / file size trade-off? We could, of course, make a separate format for Int16
and Float32
GeoTiffs?
What about generating the Int16
files for most zooms, but switching to Float32
at zoom 14/15+?
I suspect that that will confuse the GDAL WMS driver, since it treats the whole thing as a single source with overviews.
No, I tried, GDAL is fine with specifying a pixel type that is different from the data pixel type. This works:
`
`
Regarding compression: Rounding the elevation values to one decimal for zoom 14, maybe 2 for zoom 15+ will decrease size quite a bit. I think this would be the better solution than switching to cm and int32.
@flitzi different pixel types at varying zooms? That's fantastic.
GeoTIFFs are currently
Int16
, which means that the limit on vertical resolution is 1m. Higher resolution datasets (i.e. LiDAR) effectively lose resolution when converted toInt16
. (NED is distributed asFloat32
, so this may already be occurring.)