tilezen / joerd

Joerd can be used to download, merge and generate tiles from digital elevation data
MIT License
315 stars 49 forks source link

Elevation data isn't normalized #96

Open jan-tosovsky-cz opened 7 years ago

jan-tosovsky-cz commented 7 years ago

I expected your elevation service offers normalized data without peaks. However, some tiles contains strange values, which e.g. disallow reliable determining the min/max range.

See e.g. https://s3.amazonaws.com/elevation-tiles-prod/terrarium/12/2194/1409.png There is rgb(255,255,0) value at X=169, Y=167 which returns height of 32767 meters (!)

I accept the service is offered for free so one cannot expect hight quality data, but it IMHO limits the use in real apps.

kevinkreiser commented 7 years ago

are you sure it isnt -32768? if it is, you have found a pixel with a FILL_VALUE value as described here: https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl1_v003

this would mean there was no data for this location. what a lot of applications do when using this data is apply a bilinear filter when computing the a value at a given location. when doing so they will ignore those pixels with this FILL_VALUE and give portion of the weighting coefficients to the pixels that did have data. for example this bit of code works with the hgt format of this same data and ignores FILL_VALUE pixels: https://github.com/valhalla/skadi/blob/master/src/skadi/sample.cc#L107-L153

i'm not sure if there will be an effort to fill in the fill value pixels based on neighboring pixels but it is also something that we might want to do in the future. others can weigh in if that may or may not be the case.

jan-tosovsky-cz commented 7 years ago

-32768 would be rgb(255,255,255), calculated using this equation: (256 * r) + (g) + (b / 256) - 32768.

Edit: I overlooked minus sign, in that case it would have to be rgb(0,0,0).

jan-tosovsky-cz commented 7 years ago

Btw, SRTM data is offered in 'Void Filled' form https://lta.cr.usgs.gov/SRTMVF

jan-tosovsky-cz commented 7 years ago

The practical impact can be seen e.g. here (not exact place as above). It ruins auto-exposure feature: https://tangrams.github.io/heightmapper/#12/48.8192/12.8902 (not somewhere in the ocean, but in Germany)

zerebubuth commented 7 years ago

That looks like a bug, especially as the worst artefact disappears at zoom 13. Looking at the tile, there are several spurious peaks in the green channel and the one you mention in the red and green channels. These seem to be sited around the boundaries of a void-filled area. Below is the blue channel with artefact positions highlighted in red, which are near areas of significantly less "noise" than elsewhere and therefore highly likely to be filled voids.

void-boundary-artefacts

I suspect that the artefacts may be due to interaction between the super- and sub-sampling filters and data smoothness discontinuities at the filled void boundary, but I haven't confirmed that. If so, this is related to #41 but harder to detect the boundary as it's internal to the dataset.

Btw, SRTM data is offered in 'Void Filled' form https://lta.cr.usgs.gov/SRTMVF

We are using SRTM version 3.0 global 1 arc-second where available, including over Germany. This data set has been void-filled, as can be seen from the image above.

I accept the service is offered for free so one cannot expect hight quality data, but it IMHO limits the use in real apps.

We are trying to use the best quality open data available for each region. SRTM is a very valuable "basic" data set which covers most of the world, but many countries and regions make higher quality data available. Various sources have been submitted for inclusion which will improve the quality in the future.

I think we can make high-quality free and open data. This version isn't perfect, but we thought it was a good place to start and the next version will be better.

Until then, one work-around might be to normalise the height to the 1-99 percentile range, which will avoid the range being skewed by small numbers of outliers.