Normal format (transformed) PNGs are fuzzy and muted

[nvkelso]

Looks like MarbleTiler fork is producing different normals than Joerd. There are probably more than one way of calculating these, but the output of Joerd is better for presentation cartography like Walkabout style and MarbleTiler should better match that output.

Specifically, MarbleTiler dev (top again) seems to have less exaggeration on the terrain / less details (compared to joerd prod)? Is the kernel size larger in MarbleTiler versus Joerd? That may account for more generalized looking relief.

• MarbleTiler: https://github.com/mojodna/marblecutter/blob/mapzen/marblecutter/transformations/normal.py
• Joerd: https://github.com/tilezen/joerd/blob/master/joerd/output/normal.py

Example 1:

• https://d34o1q5y1d9is8.cloudfront.net/normal/6/33/22.png
• http://tile.mapzen.com/mapzen/terrain/v1/normal/6/33/22.png (requires API key)

marbletiler fuzzy, less exaggeration joerd crispier (more detail), more exaggeration

Example 2:

• https://d34o1q5y1d9is8.cloudfront.net/normal/10/163/395.png
• http://tile.mapzen.com/mapzen/terrain/v1/normal/10/163/395.png (requires API key)

marbletiler joerd

Walkabout showing Example 2:

marbletiler joerd

[zerebubuth]

I'm not sure exactly what's going on here. The normal calculation looks identical in both, so I tried to check whether it's the underlying data which is "fuzzy". Looking at the terrarium tiles:

• https://d34o1q5y1d9is8.cloudfront.net/terrarium/6/33/22.png
• http://tile.mapzen.com/mapzen/terrain/v1/terrarium/6/33/22.png (+ API key)

Marbletiler

Joerd

The first thing is that the marbletiler version has no blue channel - the fractional sub-metre channel. The code looks fine, so it seems possible that the data might have no sub-metre resolution, which could contribute to the lack of definition. Although probably in a very minor way.

The real reason for looking at the terrarium channel is to see if that's already "fuzzy", which we can do by looking at the (stretched) red channel:

Metatiler

Joerd

It looks like the Metatiler version is already "fuzzy" and it's not the output transformation which is causing this. There are a couple of reasons why this could be:

1. Resampling. The make_vrt.sh script appears to control the resampling and defaults to near. I can't find anywhere in our notes where that's overridden, so I assume that's what this was run with. However, I would expect nearest neighbour sampling to produce a noisier rather than smoother image when downsampling, and I don't see any evidence of that here. (Nearest neighbour downsampling of SRTM would explain the lack of sub-metre height values.)
2. Source image resolution. I'm not sure I've understood the metatiler code completely - is there any chance that it might be using a low-resolution DEM at this zoom rather than downsampling a more detailed DEM?

[mojodna]

make_vrt.sh and the resulting VRTs are no longer used (I should remove the script). Resampling using lanczos (and near for masks).

Pretty sure that the root cause is source data resolution, especially given the lack of sub-meter data in the terrarium tiles.

[zerebubuth]

make_vrt.sh and the resulting VRTs are no longer used (I should remove the script). Resampling using lanczos (and near for masks).

Sorry, you're absolutely right - I was getting confused trying to navigate the source code.

Looking for lanczos, I see it's used in two places: bin/transcode.sh when creating the overviews, and read_window (in __init__.py) when reading the data. It looks like overview creation will always be downsampling, and read_window will up- or downsample as appropriate.

I've found that Lanczos can be quite unstable as a downsampling filter, although with a tendency to "oversharpen" the image rather than blur it. It especially seemed to have issues at NODATA boundaries.

Pretty sure that the root cause is source data resolution, especially given the lack of sub-meter data in the terrarium tiles.

Apologies if I'm not understanding you correctly, but do you mean that SRTM isn't part of the source data for this tile?

[mojodna]

Everything I know about lanczos I learned from @smathermather. Mind weighing in?

Apologies if I'm not understanding you correctly, but do you mean that SRTM isn't part of the source data for this tile?

I haven't checked yet, but yes, I suspect that may be the cause.

[smathermather]

GDAL's implementation of Lanczos effectively doesn't have a nodata strategy, but as long as you avoid areas with nodata, I find it very effective. It's my goto as it doesn't perform the sin of moving the peaks and valleys of the data.

What it doesn't do, unlike it's siblings, is blur data -- it retains as much fidelity as possible upon resample. It does this because it reconstructs the 2.5D surface first, then samples to the new resolution (this is also why it doesn't move the peaks and valleys, unlike average, nn, and others). I think this is what Matt is seeing when it oversharpens.

For this, you have to down sample and then upsample, e.g.: https://smathermather.com/2014/11/22/landscape-position-using-gdal/

It's possible to explicitly bake terrain generalization into this too. I haven't had time to turn this into a proper utility, but the basic framework is here: https://smathermather.com/2017/01/30/gorilla-research-in-musanze-rwanda-hillshades-continued/

Cheers.

[smathermather]

Moar:

• Ungeneralized terrain:

• Generalized terrain:

• Generalized terrain with HDR rendering and less ridiculous colors:

[nvkelso]

Closing in favor of https://github.com/mojodna/marblecutter/issues/4.