Serbian Language Script (sr)

AndyAnds265 commented 1 month ago

Description

At the moment, we are not optimized for the Serbian language. Serbian language users are currently experiencing an inconsistent and odd UI experience.

There are about 50k current Serbian users on Android so this is not an insignificant group. It has even been brought up in other areas. (https://github.com/brave/brave-browser/issues/35417#issuecomment-2151009766)

Basically here is the background and issue explained:

the Serbian language uses two separate scripts Latin and Cyrillic
currently if you set your Android device language to Serbian (Cyrillic), then you will see both Cyrillic and Latin scripts within the same UI
currently, if you set your Android device language to Serbian (Latin) then you will see mostly ENGLISH and only some Serbian with Latin Script
Meanwhile, Chromium supports both scripts (link)
it will take a little work but we can easily write a piece of code that converts between the two scripts. So there won't be any localization costs incurred. We can continue localizing in Serbian-Latin as we have been; however convert it to Cyrillic to support both or vice versa

Screenshot_20240715-212754

Steps to reproduce

Change Android device language (in settings) to Serbian (Latin) or Serbian (Cyrillic)
Delete the Brave app
Re-install the Brave app
Launch app

Actual result

See mixed script usage Screenshot_20240715-212140

Expected result

it should be 1 consistent script

Reproduces how often

Easily reproduced

Brave version

1.68 and previous *apparently this has been an issue for years

Device

Brand/model: Pixel 4a
Android version: 1.67

Channel information

[X] release (stable)
[ ] beta
[ ] nightly

Reproducibility

[ ] with Brave Shields disabled
[ ] with Brave Rewards disabled
[ ] in the latest version of Chrome

Miscellaneous information

No response

AndyAnds265 commented 1 month ago

Serbian (Cyrillic) device setting

https://github.com/user-attachments/assets/907d51cb-5998-4741-8a8b-ff34a270d8ee

AndyAnds265 commented 1 month ago

Serbian (Latin) device setting

https://github.com/user-attachments/assets/5f72d427-0692-408b-bac7-61d42ecff748

AndyAnds265 commented 1 month ago

In order for us to fix this issue, we need to:

create 2 Serbian languages on our Android app
to make things simple we can align with Chromium (and Play) we can use the same locale codes. For Cyrillic, it should just be sr. For Latin, it should be sr-Latn. In doing this, we'll be able to serve the correct script per language. That'll be the starting point.
Once we have done that, we'll need to align our translation management tool (currently Transifex but soon starting migration to Crowdin). I can create 2 Serbian languages there.
The Latin based script we can launch immediately as we have been localizing Serbian into the Latin script. So that locale will effectively be ready to go immediately.
For the sr locale, we'll need to convert the entire string base using an alphabet swapping script. I've created one before and should be able to do this relatively easily working with a dev. Changing the alphabet is literally just a swap of characters. The words themselves do not change

AndyAnds265 commented 1 month ago

Here is a dictionary of the Serbian alphabets, we can likely leverage the 4th dictionary to convert characters: https://docs.google.com/document/d/16RSD7ut05e4vtjZFQogKMB8_31xeUslZ0VBkn_iCbQ4/edit

Susexe commented 3 weeks ago

For the sr locale, we'll need to convert the entire string base using an alphabet swapping script. I've created one before and should be able to do this relatively easily working with a dev. Changing the alphabet is literally just a swap of characters. The words themselves do not change

Sadly, this is not as easy as you think. Just by going on Brave Search and setting the display language to Serbian (Cyrillic) I can already see plenty of errors. There are just too many exceptions to cover, which is not possible by using transliteration scripts. For instance, your script currently transliterates the digraphs LJ and NJ to ЉЈ and ЊЈ instead of Љ and Њ. It's probably the same for DŽ → ЏЈ instead of Џ. Sometimes these letters, one after another, are used separately, which your script wouldn't know. There are some efforts to overcome this with exception dictionaries, but they're not reliable. Once we include English words and acronyms into the picture, it's even a bigger mess (e.g., AI should be left as is, not АИ).

This is exactly the reason why I offered to help: Serbian Latin to Cyrillic transliteration is never fully correct, while Cyrillic to Latin 100% is. The real solution is to switch to Serbian Cyrillic as the main script. To accomplish this, after machine transliteration, human review is necessary. After that, everything would be fully automated.

AndyAnds265 commented 3 weeks ago

@Susexe thanks for the clarification. We're going to be switching to Serbian Cyrillic as our base language.

How could you help in this situation for example?

Susexe commented 2 weeks ago

@AndyAnds265 To start, I could review and correct the transliterated strings you processed with your script (including Brave Search and possibly the Android app, once it's completed). After that, I can assist in improving the overall translation quality, which currently needs some work. For now, however, the priority is to fix the strings and establish a solid base language for Cyrillic-to-Latin conversion.

brave / brave-browser