Add pmap:script - Githubissues

wipfli commented 4 months ago

Implements #253

wipfli commented 4 months ago

PMTiles

https://pub-726b01260c98468a9387cc0dfcb7386b.r2.dev/protomaps-europe.pmtiles with coverage for Europe

Style Localized to English

"text-field": "{name:en}" in the country labels layer

And in the places_locality layer I used this:

"text-field": [
  "format",
  [
      "case",
      ["!", ["has", "pmap:script"]],
      ["coalesce", ["get", "name:en"], ["get", "name:de"], ["get", "name"], ""],
      ["has", "pmap:script"],
      ["coalesce", ["get", "name:en"], ["get", "name:de"], ""],
      ""
  ],
  {},
  "\n",
  {},
  [
      "case",
      ["has", "pmap:script"],
      ["get", "name"],
      ""
  ],
  {}
]

nvkelso commented 4 months ago

Nice, the demo looks great!

wipfli commented 4 months ago

Style localized to Greek

"text-field": "{name:el}" in the country labels layer

And in the places_locality layer I used this:

"text-field": [
  "format",
  [
      "case",
      ["==", ["get", "pmap:script"], "GREEK"],
      ["coalesce", ["get", "name:el"], ["get", "name"], ""],
      ["!=", ["get", "pmap:script"], "GREEK"],
      ["coalesce", ["get", "name:el"], ""],
      ""
  ],
  {},
  "\n",
  {},
  [
      "case",
      ["!=", ["get", "pmap:script"], "GREEK"],
      ["get", "name"],
      ""
  ],
  {}
]

Athens

https://pub-cf7f11e26ace447db8f7215b61ac0eae.r2.dev/el.html#map=8.55/37.9815/23.801

Skopie

https://pub-cf7f11e26ace447db8f7215b61ac0eae.r2.dev/el.html#map=10.68/41.9835/21.5161

Berlin

https://pub-cf7f11e26ace447db8f7215b61ac0eae.r2.dev/el.html#map=9.54/52.5462/13.4908

wipfli commented 4 months ago

Style localized to Bulgarian

"text-field": "{name:bg}" in the country labels layer

And in the places_locality layer I used this:

"text-field": [
  "format",
  [
      "case",
      ["==", ["get", "pmap:script"], "CYRILLIC"],
      ["coalesce", ["get", "name:bg"], ["get", "name:ru"], ["get", "name:uk"], ["get", "name:sr"], ["get", "name:mk"], ["get", "name:be"], ["get", "name:kk"], ["get", "name:ky"], ["get", "name"], ""],
      ["!=", ["get", "pmap:script"], "CYRILLIC"],
      ["coalesce", ["get", "name:bg"], ["get", "name:ru"], ["get", "name:uk"], ["get", "name:sr"], ["get", "name:mk"], ["get", "name:be"], ["get", "name:kk"], ["get", "name:ky"], ""],
      ""
  ],
  {},
  "\n",
  {},
  [
      "case",
      ["!=", ["get", "pmap:script"], "CYRILLIC"],
      ["get", "name"],
      ""
  ],
  {}
],

Athens

https://pub-cf7f11e26ace447db8f7215b61ac0eae.r2.dev/bg.html#map=8.55/37.9815/23.801

Skopie

https://pub-cf7f11e26ace447db8f7215b61ac0eae.r2.dev/bg.html#map=10.68/41.9835/21.5161

Berlin

https://pub-cf7f11e26ace447db8f7215b61ac0eae.r2.dev/bg.html#map=9.54/52.5462/13.4908

bdon commented 4 months ago

I wonder if this:

["!=", ["get", "pmap:script"], "CYRILLIC"],

Is there some way to move the logic of what language uses what script into the tiles? such as:

["!=", ["get", "pmap:script"], ["get","pmap:...."]],

Otherwise we'd need to encode a map of bg -> CYRILLIC in the frontend code, right?

Is there a standardized script code instead of LATIN, CYRILLIC somewhere in Harfbuzz?

wipfli commented 4 months ago

Is there a standardized script code instead of LATIN, CYRILLIC somewhere in Harfbuzz?

We could use the ISO15924 script codes consisting of 4 letters, see https://unicode.org/iso15924/iso15924-codes.html. Harfbuzz uses those as well. Note however that the java function Character.UnicodeScript.of already uses the standardized English names of the scripts from unicode, see https://docs.oracle.com/javase%2F8%2Fdocs%2Fapi%2F%2F/java/lang/Character.UnicodeScript.html

Is there some way to move the logic of what language uses what script into the tiles?

I think it makes most sense to decide in the style json what script can be used for a given language. The script is a characteristic of the language and not of the individual feature in a vector tile...

Below you find an attempt to group languages by script. Note that some languages can be written in more than one script, e.g., Malay can be written in Latin, Arabic, and Thai.

Scripts Supported by MapLibre

Latin

AFRIKAANS
ALBANIAN
AZERBAIJANI - also cyrillic, arabic
BASQUE
BOSNIAN - also cyrillic
CATALAN
CROATIAN
CZECH
DANISH
DUTCH
ENGLISH
ENGLISH (AUSTRALIAN)
ENGLISH (GREAT BRITAIN)
ESTONIAN
FINNISH
FILIPINO
FRENCH
FRENCH (CANADA)
GALICIAN
GERMAN
HUNGARIAN
ICELANDIC
INDONESIAN
ITALIAN
LATVIAN
LITHUANIAN
MALAY - also arabic, thai
NORWEGIAN
POLISH
PORTUGUESE
PORTUGUESE (BRAZIL)
PORTUGUESE (PORTUGAL)
ROMANIAN
SLOVAK - also cyrillic
SLOVENIAN
SPANISH
SPANISH (LATIN AMERICA)
SWAHILI
SWEDISH
TURKISH
UZBEK - also cyrillic, arabic
VIETNAMESE
ZULU

Arabic

ARABIC
FARSI
URDU

Cyrillic

BELARUSIAN
BULGARIAN - also latin
KAZAKH - also latin, arabic
KYRGYZ - also arabic
MACEDONIAN
RUSSIAN
SERBIAN - also latin
UKRAINIAN

Han ?

CHINESE
CHINESE (SIMPLIFIED)
CHINESE (HONG KONG)
CHINESE (TRADITIONAL)

One Language Per Script

AMHARIC
ARMENIAN
KOREAN
JAPANESE
GEORGIAN
GREEK
MONGOLIAN - also cyrillic

Scripts Not Supported by MapLibre

Devanagari

GUJARATI
HINDI
MARATHI
NEPALI

One Language Per Script

HEBREW
KANNADA
BENGALI
BURMESE
KHMER
LAO
MALAYALAM
PUNJABI
SINHALESE
TAMIL
TELUGU
THAI

wipfli commented 3 months ago

I did some experimentation with the normal protomaps tiles and the normal protomaps style, but added a javascript hook to MapLibre GL JS which allows you to modify vector tile feature properties in the client.

https://wipfli.github.io/maplibre-feature-properties-transform-example/language?script=Latin&language=fr#map=8/38/24

You can choose different languages. The logic is contained here and uses some regular expressions on the name tag.

The main problem I see at the moment is when the name tag contains text in multiple scripts. For example, OSM Relation 11118019 has

name = Ngushtica e Otrantos / Canale d'Otranto / Στενό του Οτράντο

So it contains text in Latin and Greek.

wipfli commented 3 months ago

... and I delete text from the name name tag which has the same script as the chosen language. For example, when the language is set to German, the result is this:

https://wipfli.github.io/maplibre-feature-properties-transform-example/language?script=Latin&language=de#map=8/39.996/19.233

nvkelso commented 3 months ago

This isn't working for me in safari or chrome on iOS.

wipfli commented 3 months ago

Looks like also firefox does not like the syntax data:,<your-js-code> for dataURL containing javascript. I replaced it now with

const decodedString = decodeURIComponent(`${settings}${transform}`);
const blob = new Blob([decodedString], { type: 'application/javascript' });
const workerUrl = URL.createObjectURL(blob);
maplibregl.importScriptInWorkers(workerUrl);

wipfli commented 3 months ago

Sorry for getting side-tracked by the MapLibre GL JS feature properties transform thing. I think for now it would be best to move on with this pull request as is.

Do you want me to shorten things like LATIN to LATN as defined here? I am against it because tile size savings are minimal due to string indexing in MVT and the readability gets worse.
Do I need to add some tests?

wipfli commented 3 months ago

I am a bit lost in spotless:apply.

wipfli commented 3 months ago

@bdon this is ready for review from my side.

wipfli commented 3 months ago

Thanks for the reviews! I think I addressed all comments. How should I handle the failing sonarcloud CI?

sonarcloud[bot] commented 3 months ago

Quality Gate failed

Failed conditions
64.5% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

protomaps / basemaps

Add pmap:script #254

PMTiles

Style Localized to English

Athens

Skopie

Berlin

Style localized to Greek

Athens

Skopie

Berlin

Style localized to Bulgarian

Athens

Skopie

Berlin

Scripts Supported by MapLibre

Latin

Arabic

Cyrillic

Han ?

One Language Per Script

Scripts Not Supported by MapLibre

Devanagari

One Language Per Script

Quality Gate failed