search

tilezen / vector-datasource

Tilezen vector tile service - OpenStreetMap data in several formats
https://www.nextzen.org/
Other
506 stars 119 forks source link

Support alternate points of view in boundaries layer #1552

Closed nvkelso closed 5 years ago

nvkelso commented 6 years ago

Natural Earth supports the concept of "points of view" (POV) to indicate the claimants for different boundary disputes.

Right now Tilezen supports just the default "defacto" point of view, but sometimes that deviates from US State Department and United Nations views. And showing maps of India, Pakistan, China, and other areas require special treatments.

Tilezen should support both a default POV (now) and additional POV for country and disputed boundary lines (possibly also region but Natural Earth data doesn't support that now).

musculman commented 6 years ago

When looking at this feature it would probably make sense to think about how to solve the fact that you would need to give different geometries depending on the view used.

In particular, we should not only think about a particular border being listed as belonging to two (or more) countries, but also the fact that these countries might have different understanding on where the actual border is.

Therefore, it has two main use cases:

For some simple raster tile examples, I think it might be worth looking at the example from HERE Map Tile API.

Looking at a possible implementation for a border with an example, could be:

Additional attributes worth considering are:

zerebubuth commented 6 years ago

As an alternative way of looking at things; borders (national or otherwise) are largely fictitious. In some cases, there is a physical barrier marking the boundary. This makes it easy to map and draw - the big fence or wall is the border. Where there is no physical barrier, the border is by nature more abstract and porous, so it matters less exactly where it is drawn.

Some countries mandate, sometimes by law, that maps used within their jurisdiction are drawn to support their opinions (and often, ideology). Therefore, rather than calling it a view, we could call it a mandatory_delusion where it differs from the ground truth (de facto) feature.

The logic for drawing might look something like:

ground_truth = feature.properties.get('ground_truth', 'draw')
key = "mandatory_delusion:" + this_country_code
should_draw = feature.properties.get(key, ground_truth)
if should_draw = 'draw':
  # draw it
# otherwise, don't draw it

In other words, a country-specific flag overrides a default "ground truth" flag, which defaults to drawing. If it turns out there are large blocks of similar delusions, we can do a two-level scheme (i.e: look for the country first and then the block for the override). This way we don't have to put a lot of extra properties on the large numbers of borders or labels which are not disputed.

musculman commented 6 years ago

I would really not call it something like mandatory_delusion or ground_truth. By the pure nature of this topic, it is highly sensitive and controversial, using property names like those can only cause problems. The word view derives from the fact that what you are seeing when using that view is just the world according to certain political understanding of a particular country, in other words, the political view of that country. It's fine if another word is used, but if you look at other companies providing this solution, you will notice that the choice of words is always carefully considered.

Regarding the fact that it should not bloat non disputed borders, that is usually achieved by having a simple boolean that will tell you is_disputed= true|false and only if it is true you look for the additional optional values, like shown above. That way you just need a single value that can be nicely encoded.

Regarding the ground truth, please keep in mind, there is no such thing, only the truth according to certain political considerations, so the default view is in fact already an agreement between one or more countries. You could consider for example having the default view as the UN view, but even in that case, the UN considers many borders as disputed and there are many lines of control that further modify it, as I mentioned above. One should be very clear about what is their default view, whether it is UN, US or any other variety.

zerebubuth commented 6 years ago

Regarding the ground truth, please keep in mind, there is no such thing, only the truth according to certain political considerations

You're right, which is why (half-jokingly, although I should have telegraphed that more clearly :bowing_man: ) I suggested we make this explicit in the property name. In many cases, there is physical infrastructure which can be measured as empirical truth, but I imagine the vast majority of borders are not marked physically in any way. I suppose that proves the point that people can be quite sensitive about being reminded of the (at best, consensual) unreality of borders.

having a simple boolean that will tell you is_disputed= true|false

There's no need for an additional boolean under the scheme I described, the presence of one or more view:${ISO_CODE} keys implies is_disputed=true and the absence of any implies false.

musculman commented 6 years ago

Thanks for the additional clarification. Let me see if I understand that proposal correctly, because I think I am missing one point there.

In the case I mentioned in my first comment, of actual changes on the country border depending on the view; let's consider a border that only exists for country A and let's call that border X and every other country in the world puts that border somewhere else, let's call it border Y. Remember they are different geometries. With the system you describe, how would you encode that? Border X would have view:iso_of_country_A and nothing more but what would Border Y have? All other possible countries? or would it simply not mention any disputed? If you do that then the view of country A would also show it, that is why above I mentioned this mechanism of is_accepted_by and is_disputed_by... I know it is probably confusing so that's why I am happy to see a discussion to improve it / design something else that is better. Let me know please if I misunderstood your proposal or how to code such an example. Just for the record, it is a real life example but I did not use real country names :) :)

zerebubuth commented 6 years ago

I'm suggesting that it could be described as:

(If we don't like show|hide then we could use accepts|disputes or True|False. I think show|hide would make more sense to people writing the map style, but any binary pair of values would work.)

It's possible to work out is_disputed by asking if any key starting in view: is show. We don't need to list the countries which follow the majority view - since there are only two options, they must hold the opposite view to the minority.

So Country A will show Border X and hide Border Y, because those are the explicit overrides. Any other country will show Border Y and hide Border X, because those are the defaults.

For more complex situations, where several countries hold a different view, the scheme still works although it may be wasteful to list them all out. The maximum number of different disputants that I can see is a six-way dispute over the Spratly Islands. However, there's very little common geometry amongst the claims, so I'd assume we'd still end up with a single view:XX property per feature.

musculman commented 6 years ago

Thank you for the explanation, I had indeed misunderstood it :) For examples like the one you listed we might need to think of having also the additional properties like claimed_by or controlled_by depending on the routing capabilities that you want to support, if any, because you can usually ask your routing engine to avoid areas controlled by certain countries or to avoid disputed areas at all. For example if your user has a certain passport that is not allowed to cross some countries/borders.

musculman commented 6 years ago

I will try to create a first draft of what we have discussed on this, unless someone is already preparing it.

nvkelso commented 6 years ago

Suggest using "point of view" like we do language localization but instead by ISO 2-char country codes (in practice this is sparsely populated on minority of features, but then some features are very contentious!):

Alternatively we could support an array of "point of view" values, like:

We already have different boundary layer kind values (list) for the following, and the "point of view" can be applied equally to all of them:

With respect to "line of control":

nvkelso commented 6 years ago

I'm open to {"point_of_view:cc":"show"} or {"point_of_view:cc":"hide"}, too.

We could in that case still keep it "truthy" but make it True (show), False (hide), and Null (no opinion/use default point of view, drop the property)?

zerebubuth commented 6 years ago

I would prefer {"point_of_view:cc": "show"} or {"point_of_view:cc": "hide"}, because I think this is more readable than boolean values. If we really, really want to use default values, then I think we should use:

If the property isn't given (i.e: None), then the view of the country is that of the default majority, opposite to that of the explicitly listed minority.

musculman commented 6 years ago

@nvkelso: I have some questions about your last comment:

Boundaries Layer

(...)

Natural Earth supports the concept of points of view (POV) to indicate the claimants for different boundary disputes. When displaying a map you need to consider if you are going to support the different points of view or if you are just going to use the de-facto one, which is in line with US State Department and United Nations views (TODO: link?). If a particular point of view is used, the boundaries of the selected country view are shown; otherwise the default boundaries view is shown. A typical use case is that a particular area of the world is considered as disputed by two or more countries. Therefore, each country shows that area as part of itself. If you request a particular tile with the political view of country A, the response shows the area as belonging to A. Similarly, if you request the tile with the political view of country B, the response shows the area as belonging to B. This means that in order to use this optional feature you need to define a point of view value, in a similar way as you define language localization, but using ISO 2-char country code. In each case, in order to display correctly the disputed borders you need to understand what border should be displayed in each case, and control this logic on your Rendering Engine.

Please note that the the boundary layer kind line_of_control is also relevant for displaying correctly the point of view.

Border X properties: `{"view:iso_A": "true"}`
Border Y properties: `{"view:iso_A": "false"}`

Country A will show Border X and hide Border Y, because those are the explicit overrides. Any other country will show Border Y and hide Border X, because those are the defaults. In case of borders that have different geometry depending on the point of view used, they will have view:iso_C and only the country with that point of view, C, in this case, will display the new geometry as everybody else will simply ignore it. It's possible to work out what borders are disputed by asking if any key starting in view: is true. We don't need to list the countries which follow the majority view - since there are only two options, they must hold the opposite view to the minority.

Boundaries properties (optional):

(...)

Boundary kind_detail values for country

For kind country we can have the following optional kind_detail values:

nvkelso commented 6 years ago

Something like?

For undisputed country boundaries:

For disputed country boundaries where default needs to be hidden and different geometry shown:

Then Tangram would set (psuedocode):

layer:
    boundaries:
        disputed:
            filter: { point_of_view: True }
            draw:
                line:
                    // hide lines with points of view unless that country believes it's theirs
                    visible: function() { return feature['point_of_view:'+global.point_of_view_cc]) || false; }

I think claimed_by and controlled_by are useful for the backend database but shouldn't be exported in the vector tiles (instead transform them into point_of_view:cc).

I can take care of the documentation parts if this seems workable.

I'm curious if @bcamper thinks the above solution is best situation for Tangram.

musculman commented 5 years ago

@nvkelso are you still planning to do the documentation part? Otherwise I can follow with a PR on this. But, because my latest proposal on my comment is slightly different compared to your last comment, it is perhaps better that you do it? Just let me know, thank you.

nvkelso commented 5 years ago

I can take care of the PR, once back in the office next week from Thanksgiving break.

musculman commented 5 years ago

No rush! Just wanted to know if you were going to take care of it at some point or should I go ahead. Enjoy your days off!

nvkelso commented 5 years ago

I've started prototyping this out...

nvkelso commented 5 years ago

After prototyping this out a bit in January 2019, I now propose (v2) to "localize" the kind like kind:iso instead of introducing extra / duplicate spagetti lines, for the ~20 TZ languages / countries, and other major claimants (like Morocco, Pakistan, and Palestine), just like we do with name:* today. Meaning we should not see any significant file size increase from this approach over v1.6 data model.

This also addresses an issue where the kind of the feature changes per point of view, not just the visibility of the line (as was the focus of the earlier proposal above).

Similar to the last proposal, this proposal does not address the issue that some labels in the places layer (country and region primarily) should also be disabled / changed in parallel. For example, India's Arunachal Pradesh region is claimed by China as part of the Tibet region and ideally there would be a way to mark that place's localized kind:* to unrecognized as well, but that's out of scope for v1.7 (though consistent with this proposal).

Prototype data for Natural Earth v5.0.0-pre1:

Tilezen considerations:

  - &ne_country_boundaries_kind
    lookup:
      key: { col: featurecla }
      op: '=='
      table:
        - [ 'Disputed (please verify)', 'disputed' ]
        - [ 'Indefinite (please verify)', 'indefinite' ]
        - [ 'Indeterminant frontier', 'indeterminate' ]
        - [ 'International boundary (verify)', 'country' ]
        - [ 'Lease limit', 'lease_limit' ]
        - [ 'Line of control (please verify)', 'line_of_control' ]
        - [ 'Overlay limit', 'overlay_limit' ]
        - [ 'Unrecognized', 'unrecognized' ]
        - [ 'Map unit boundary', 'map_unit' ]
        - [ 'Breakaway', 'disputed_breakaway' ]
        - [ 'Claim boundary', 'disputed_claim' ]
        - [ 'Elusive frontier', 'disputed_elusive' ]
        - [ 'Reference line', 'disputed_reference_line' ]
      default: null
global:
  - &ne_localized_kind_properties
    'kind:iso': {col: FCLASS_ISO }
    'kind:us': {col: FCLASS_US }
    'kind:fr': {col: FCLASS_FR }
    'kind:ru': {col: FCLASS_RU }
    'kind:es': {col: FCLASS_ES }
    'kind:cn': {col: FCLASS_CN }
    'kind:tw': {col: FCLASS_TW }
    'kind:in': {col: FCLASS_IN }
    'kind:np': {col: FCLASS_NP }
    'kind:pk': {col: FCLASS_PK }
    'kind:de': {col: FCLASS_DE }
    'kind:gb': {col: FCLASS_GB }
    'kind:br': {col: FCLASS_BR }
    'kind:il': {col: FCLASS_IL }
    'kind:ps': {col: FCLASS_PS }
    'kind:sa': {col: FCLASS_SA }
    'kind:eg': {col: FCLASS_EG }
    'kind:ma': {col: FCLASS_MA }
    'kind:pt': {col: FCLASS_PT }
    'kind:ar': {col: FCLASS_AR }
    'kind:jp': {col: FCLASS_JP }
    'kind:ko': {col: FCLASS_KO }
    'kind:vn': {col: FCLASS_VN }
    'kind:tr': {col: FCLASS_TR }
    'kind:id': {col: FCLASS_ID }
    'kind:pl': {col: FCLASS_PL }
    'kind:gr': {col: FCLASS_GR }
    'kind:it': {col: FCLASS_IT }
    'kind:nl': {col: FCLASS_NL }
    'kind:se': {col: FCLASS_SE }
    'kind:bd': {col: FCLASS_BD }

Example POV:

_In QGIS these are accomplished by a coalesce( "FCLASS_CN", "FEATURECLA" ) to prefer the localized kind, but fallback to the default kind on all the 3 boundaries SHP themes. The polygons are also built using similar attributes and new build targets – but since we don't include admin polygons in Tilezen those are not included in the prototype data archive above._

China:

screen shot 2019-01-29 at 22 46 09

India:

screen shot 2019-01-29 at 22 46 20

Pakistan:

screen shot 2019-01-29 at 22 46 31

Default:

screen shot 2019-01-29 at 22 46 42

Example data table attributes

_See featurecla column at far left compared to fclass_cn column at far right._

screen shot 2019-01-29 at 23 00 22 screen shot 2019-01-29 at 22 58 53
zerebubuth commented 5 years ago

Started work on this in #1809 (not a real PR - just putting it there so it's easier to discuss). Here's a view of tile 5/22/12 showing India's POV with the kind (not kind:in - that's country for all the lines shown here) as:

image

nvkelso commented 5 years ago

Looks like we're sometimes missing the POV kind values, here for China:

image

Versus what was in the shapefile delivery from Natural Earth v5.0.0-pre1:

image

@zerebubuth can you have a look, please?

nvkelso commented 5 years ago

But otherwise this is working as expected :)

There's also work to add this in from the Natural Earth 1:50M and 1:110M scale sets... and from populated_places and states_provinces as some country and region capitals aren't recognized, and some regions aren't recognized. That data still needs to be created.

screen shot 2019-02-13 at 23 04 36 screen shot 2019-02-13 at 23 04 24 screen shot 2019-02-13 at 23 04 16
nvkelso commented 5 years ago

BTW, the missing China boundary at zoom 6 along Nepal is because it's NE min_zoom of 7, so that'll need to be changed in the source data.

And this funk between Israel and Palestine missing attributes upstream in NE:

image

zerebubuth commented 5 years ago

Looks like we're sometimes missing the POV kind values, here for China:

I think you might have been hovering over this line instead:

image

The version of the shapefile I have here says its default feature class is "Indefinite (please verify)" and only override is FCLASS_IN to "Unrecognized". The line to the west, which I think is the one you selected in QGIS, looks like this in tiles to me:

screen shot 2019-02-14 at 12 35 45
nvkelso commented 5 years ago

Ah, you're right! I'll update the boundary attributes upstream in Natural Earth to fill in the missing info. Thanks :)

nvkelso commented 5 years ago

This is working as expected!

For final build I'll provide v5.0.0-pre2 files which address Cyprus, Israel, and few other bug fixes.

Followup in https://github.com/tilezen/vector-datasource/issues/1840 for lower zooms, and the places layer in v1.8.

nvkelso commented 5 years ago

Fixed via https://github.com/tilezen/vector-datasource/pull/1809.

Data hotfixes in https://github.com/tilezen/vector-datasource/issues/1841.