Update road sort_key values (paired with landuse updates)

[nvkelso]

As we widen the range of landuse sort_key values in in https://github.com/mapzen/vector-datasource/issues/364 and https://github.com/mapzen/vector-datasource/issues/504, we need to offset the roads up at the same time.

Complicating this, road tunnels (and any negative layer value) sometimes conflict with landuse (explicate) and water (implied) sort keys. In New York City, part of the tunnel under the Hudson to New Jersey has layer=-2 so it's currently out of the road range into the landuse range. We need to include a healthy buffer between these ranges!

[nvkelso]

Here's the Holland Tunnel with a sort_key of 4, which is a result of it being tagged both as tunnel=yes, and as layer=-1 (Motorway = 24. Then 24 - 10 = 14 for tunnel. But finally -1 + 5 = 4 for the layer):

In previous versions of Eraser Map stylesheet, this eastern portion of the tunnel was under the water polygon, completely obscuring the tunnel feature. It only works today because we hack the server data values by adding 6 to everything.

[nvkelso]

Initial Recommendations:

This is dependent on final landuse layer sort_key changes. Let's confirm those before making any changes here. See chart attached to https://github.com/mapzen/vector-datasource/issues/364.

1. Start roads layer sort_val = 301 instead of 0 to give landuse layer sort_key values room (currently 91 values, with room to grow) & a buffer for map sandwich data visualization "underlays".*
2. Update aerialway adjustment to use 335 (was 34). See question 1 below, though.
3. Update layer=* adjustment to use 335 and 306
4. Consider supporting layer=* values ±6 and even ±7 (above numbers would adjust to all numbers above upward).

5. Set explicate sort_keys for source='naturalearthdata.com' roads:

   'Major Highway', 'Beltway, 'Bypass' 224 'Secondary Highway' 222 'Road' 221 'Track' 219 'Unknown' 217

6. Make some room for transit and water layer features to be inserted in roads between tunnels and default, and default and bridge; and reserve 300 for outline as separate from layer=-5 line. So all values subject to revision.

Question:

1. Areaways should probably always be above highways, even after layer ± sorting. (Portland, New York), though! So instead of 34 all the time, treat them like a bridge and +10 (and keep it eligible for layer ± like it is now), so baseline of (35,36,37) –> (235,236,237) in new? @zerebubuth
2. Seems like current code always assumes areaways have a layer tag, but that's not true in the NY case, and it's display is in fact broken in Eraser Map app. If there is no layer tag, should we still automatically + 5 to get it above the tallest road bridge (40,41,42) –> (240,241,242)? @zerebubuth
3. We currently see buildings incorrectly interacting with roads (Bay Bridge anchorages in SF), is something special needed to allow this to be fixed? @bcamper @zerebubuth

Constraints:

1. Current ordering sequence: Earth (at zero), then landuse, then river lines, then water fills, then water strokes, then boundary (political) lines, then buildings (zoom < 15), then roads, then boundary fences, then buildings (zoom >= 15). Labels (icons, text) are done in a separate pass.
2. Data visualization "underlays". These are often under roads, but sometimes need to be under water and even under landuse or under the background "land". We should assume that "land" is the ur landuse with the lowest sort key in the landuse range.

Research:

1) At zooms <15, sort_val are currently in the range of [14, 27] based on the type of "road" feature. The code comments say [14, 24], but we added explicate aerialway exceptions a few releases back for gondola and cable_car (27), and chair_lift (26) and other aerialway (25). aeroway values for airport taxiways fall in with secondary and tertiary roads.

2a) At zooms >=15, the sort_val has a total range of [0,39].

In these "high zooms", the basic sort_val in low and mid-zooms is modified by adding 10 if a feature is a bridge, and subtracting 10 if it's a tunnel. The sort_val for aerialway is clipped to 34 (motorway+10). So this gives us a range of [4, 34]. Then 2b) for layers kicks in.

if bridge in ('yes', 'true'):
    sort_val += 10
elif (tunnel in ('yes', 'true') or
   (railway == 'subway' and tunnel not in ('no', 'false'))):
   sort_val -= 10

2b) But there is one other zooms >=15 modification based on OSM's layer tag (which is used to "describe vertical relationships between crossing or overlapping features").

# The range of values from above is [5, 34]
# For positive layer values, we want the range to be:
# [34, 39]
if layer > 0:
    sort_val = int(layer + 34)
# For negative layer values, [0, 5]
elif layer < 0:
    sort_val = int(layer + 5)

Which results in a total range of [0,39].

NOTE: Looks like we currently only support layer values in the ±5 in our code today, which matches the wiki. But reviewing layer values on TagInfo, most values are ±5, but there are a few ±6 and even ±7.

layer   count
7   57
6   389
5   6093
4   4639
3   16714
2   87675
1   2545293
0   114641
-1  1322374
-2  37631
-3  8291
-4  5606
-5  4945
-6  39
-7  4

Raw counts:

Log scale:

• Simplistically: With reserved [0,9] and [190, 198] ranges for data visualization "underlays", and water layer features [150, 159] and boundaries layer features [180, 189], and buildings (<15) at [199]. This would also allow boundaries (fences) [250, 254] to be above roads, and buildings (>= 15) at [255] to be above all.

[nvkelso]

@zerebubuth, @rmarianski, @bcamper, @blair1618, @meetar, @sensescape to sanity check

[zerebubuth]

Make some room for transit and water layer features to be inserted in roads between tunnels and default, and default and bridge

How much room?

Areaways should probably always be above highways, even after layer ± sorting. (Portland, New York), though! So instead of 34 all the time, treat them like a bridge and +10 (and keep it eligible for layer ± like it is now), so baseline of (35,36,37) –> (235,236,237) in new?

Aerialways are currently always treated like bridges and get a +10 when zoom >= 15. The issue is that the range of values we allowed ourselves was so small that they got "clipped" to 34 always (+/- layer changes).

Seems like current code always assumes areaways have a layer tag, but that's not true in the NY case, and it's display is in fact broken in Eraser Map app. If there is no layer tag, should we still automatically + 5 to get it above the tallest road bridge (40,41,42) –> (240,241,242)?

We don't assume aerialways always have a layer tag, and in the absence of a layer tag they are treated like bridges. In what way is it broken in the Eraser Map app, and what values for sort_key would fix it?

We currently see buildings incorrectly interacting with roads (Bay Bridge anchorages in SF), is something special needed to allow this to be fixed?

What do you mean by "incorrectly interacting" - are they z-fighting? What would be the correct behaviour and the sort_key values needed to fix it?

[bcamper]

1. We currently see buildings incorrectly interacting with roads (Bay Bridge anchorages https://www.openstreetmap.org/way/236374788 in SF), is something special needed to allow this to be fixed? @bcamper https://github.com/bcamper @zerebubuth https://github.com/zerebubuth

Could you elaborate on this one please? What's the issue we're seeing now and desired behavior? Is there rendering behavior you're unclear on?

[nvkelso]

@zerebubuth

How much room? (For non-road features like water and transit in roads range)

I propose the following in the Open Office sort_key.ods.zip doc:

• 4 slots between default road sort values and the tunnels + negative layer ranges.
• 2 slots inside the default road sort values between major and minor roads.
• 3 slots between default road sort values and bridges + positive layer ranges.

We don't assume aerialways always have a layer tag... where is it broken in Eraser Map...

The New York city example shows the Roosevelt Island Tramway sorting under FDR Dr instead of above it.

My proposed value in in the same Open Office document:

sort_key.ods.zip

I'm proposing if the feature is aerialways=* and layer is not provided, then we set layer=5 before processing the offset. Are there counter examples where this would not work?

@zerebubuth and @bcamper

Buildings + Roads: What do you mean by "incorrectly interacting" - are they z-fighting? What would be the correct behaviour and the sort_key values needed to fix it?

Near the SF Mapzen offices 17/37.78870/-122.38996

The Bay Bridge highway should be flying over the anchorage buildings (they top out at the same level) and the pier buildings (the bridge is far higher than these 2 story buildings).

Have you ever run into it before?

I'm not sure what sort_key value is needed to fix this going forward, that's partly why I'm proposing space in the ranges to put buildings below or above roads (it could also vary by zoom, a common practice not yet adopted by our stylesheets).

Looking at how Eraser Map and Skin & Bones are written today, it's getting a order: 50, well above the current range for roads which would explain why buildings are above roads. But dealing with the height extrusion and the order how should this be modeled in OSM, and how should we carry thru in tiles? We had earlier said increasing the range some would be okay assuming buildings were below roads – does the proposal in sort_key.ods.zip seem like it is worth trying?

[zerebubuth]

The New York city example shows the Roosevelt Island Tramway sorting under FDR Dr instead of above it.

Ah, that's bad data - the Roosevelt Island Tramway doesn't have a layer tag, which implies layer=0, but the section of FDR Drive is tagged as layer=1. Someone needs to edit the former to add layer=2 if the tramway goes above the FDR at that point. That should resolve the issue with the current sort_key and this new one too.

The Bay Bridge highway should be flying over the anchorage buildings (they top out at the same level) and the pier buildings (the bridge is far higher than these 2 story buildings).

As I understand it (and, @bcamper please correct me if I'm wrong), all the extruded buildings have a z-depth less than any other feature. I think this means we'd need to not extrude the anchorage buildings and give them some sort_key < layer=0, bridge=yes, and then they'd display as 2D polygons under the roads...?

[nvkelso]

Ah, that's bad data - the Roosevelt Island Tramway

Cool, I'll fix the data and we'll leave that logic for aerialways, bridges, and layers as-is.

The Bay Bridge highway should be flying over the anchorage buildings we'd need to not extrude the anchorage buildings and give them some sort_key < layer=0, bridge=yes, and then they'd display as 2D polygons under the roads...?

Clever! Looks like most of those are tagged correctly in OSM for the Bay Bridge and we're passing them through (as kind:anchorage) so this could be a simple scene file change to select them and not extrude them.

• Are there any other building types we should consider removing extrusion on?
• Would it be possible to extrude the building the default amount (set in the scene file to 20), but still have them below the road? What if the road coming in was not a bridge, but the road going out was?
• Would we ever extrude roads like buildings to get them to approximate real-world heights? I remember seeing a Tangram branch with polyline extrusion, but I don't think that's supported in master today?

[zerebubuth]

Would it be possible to extrude the building the default amount (set in the scene file to 20), but still have them below the road?

It's my understanding that extruded things (things with "real" depths in the z-buffer) can't be below 2D things (things with sort_key depths in the z-buffer), in current versions of Tangram.

It should be possible to "float" roads above buildings, as it's using a 3D rendering engine under the hood. But I feel like it would take considerable work in Tangram to replicate some of the stuff we're doing with sort_key depths at the moment (e.g: casing). As I understand it, the fill is a very small fraction of a unit closer to the camera than the casing, but that would cease to work reliably if the fill/casing had real (and varying) depths. It would be z-fighting central, possibly even z-fighting II turbo. I'm sure it could be made to work using textures instead, but at the cost of a bunch of work.

[matteblair]

@zerebubuth has the gist of it. A road geometry cannot be made to draw over an extruded building geometry by changing its order because order only acts as a tiebreaker between geometries at the same height. If, however, the building is not extruded and is given an order below that of the intersecting road (as Matt suggests), then the road will indeed draw over the building polygon.

"Why does order behave so strangely?" you might wonder. The answer is that this behavior is the most consistent way we could find to represent mixed 2D and 3D geometry. It's a tricky problem! The "tiebreaker" behavior of order allows us to sort geometry on the ground plane and still have 3D geometry correctly occlude the things behind it. To make a bridge truly behave correctly with respect to extruded buildings, we would probably want to extrude each point of the bridge to a suitable height (which could be done, but requires further information from the tile server).

[hjanetzek]

Would it be possible to extrude the building the default amount (set in the scene file to 20), but still have them below the road?

Wasn't this what overlay blend-mode is for? I recall to have seen all transport platform 'shine' through buildings, which looked a bit odd for tilted view though

[bcamper]

To elaborate, in the future it would be interesting as @blair1618 says to extrude bridge geometries. But to do so we'd need more info on geometry semantics than we currently have, since we don't know where the bridge starts and ends, or more generally what the desired elevation for each point along the line is (relevant because bridges may be broken into multiple features "mid-air", and even if stored as a single feature, will get split across tiles). If the server could pre-compute either a height or simply a classification for each vertex of the bridge (e.g. "ground" or "elevated"), we could construct an extruded polyline with variable height.

[nvkelso]

@blair1618 and @zerebubuth Thanks for the background, makes sense. Works for today and tomorrow. Someday: full virtual globe. Elevation is a good downpayment to that challenge! :wink:

One final test case: is there a way to make this legit building under the bridge? If I took the height off it then it'd still be extruded the default amount of 20.

Looks like one of the common mapping apps kills that and similar buildings under bridges. The other draws them transparently / sometimes drops the heights on them, so the problem is less obvious (something that might or might work for Geraldine's design, could on other designs). Even that zero height building is still z-stacked above the bridge.

[nvkelso]

Wasn't this what overlay blend-mode is for? I recall to have seen all transport platform 'shine' through buildings, which looked a bit odd for tilted view though

@hjanetzek Good eye, and I agree that's a bug! The new sort_key (order) values I'm proposing would fix it so transport platforms would be below buildings.

We can explore overlay blend-mode on the buildings, too.

[zerebubuth]

One final test case: is there a way to make this legit building under the bridge? If I took the height off it then it'd still be extruded the default amount of 20.

We could add a post-process filter to add a property (extrude:no or something) to buildings which are intersected by road bridges where the road layer > building layer. But that might have other unexpected side effects, and we'd need to build a test-case library of other buildings which go over, under, or through the building to make sure we're doing the right thing.

But to [extrude bridge geometries] we'd need more info on geometry semantics than we currently have, since we don't know where the bridge starts and ends

Yeah, that'll be far, far future - the connectivity information is lost by the transform to OGC / PostGIS data types. Getting it back by examining the mid tables might be quite slow. Insert usual rant about NEAT tiles being the One True Solution here.

[nvkelso]

@rmarianski Can you list out where the ways with layer={6,7,-6,-7} are so we can evaluate how important those are?

[nvkelso]

Updated:

1. Move sort_key = 301 default (instead of 0, leaving 300 for client side outline)
2. Between tunnels and default roads, leave 10 slots
3. Between default major roads and minor roads, leave 10 slots
4. Stop clamping aerialway, and treat them like bridges with a +10.
5. Between default roads and bridges, leave 10 slots

[nvkelso]

@bcamper and @blair1618 Can you confirm we don't do bit packing on the order (and we have range to utilize)?

[bcamper]

Right, we can't pack the order ranges because we don't know a priori what the ranges are, and we can only examine data per-tile.

[rmarianski]

@nvkelso attached is the list of lines and polygons in the dev database that have a layer value of -7, -6, 6, or 7.

layers-6-7.tar.gz

[nvkelso]

tl;dr Current clipping to [-5,5] layer range should be kept, no change needed there.

longer version: Taking a random sampling of layers with ±6, ±7, we have mine tunnels, parking structure aisles, river, underground streams, RER railroad tunnels, public transport platforms, and amusement park rides.

Here's a particularly awesome one (parking aisles in parking structure): https://www.openstreetmap.org/edit?way=105613378#map=19/55.76562/37.84491

[meetar]

@nvkelso That openoffice doc is fascinating. Is there anywhere I can learn more about its origins?

[nvkelso]

It's me going thru the shiny new documentation for tiles and all (!?) our scene files (mostly Eraser Map, Skin & Bones, Refill) and figuring out where things lie now and where we should put them. If you see something amiss, please let me know!