Implement more advanced pluralization rules

mgralikowski commented 3 months ago

Hi,

In some languages like Polish (see more) there are more than one plural form of a single word.

E.g. person|people should be translated into osoba|osoby|osób.

More examples (and valid results):

[
    'files_count' => '[1] File|[2,4] Files|[5,*] Files',
    'person_count' => 'Person|People',
    'ball_count' => 'ball|balls',
    'coach_single' => 'coach',
]

[
  'files_count' => '[1] Plik|[2,4] Pliki|[5,*] Plików',
  'person_count' => 'Osoba|Osoby|Osób',
  'ball_count' => 'piłka|piłki|piłek',
  'coach_single' => 'trener',
]

I tried adding this prompt as an additional rule but it didn't work because it conflicts in some way with the main rule. Anyway, it is not a case only for Polish so IMHO this rule should be global to make it work for all possible languages.

Tested using the claude-3-sonnet-20240229 model.

kargnas commented 3 months ago

Thank you. I also changed the code to support this kind of pluralization for other languages like Chinese, Arabic and Russian.

https://github.com/kargnas/laravel-ai-translator/compare/v1.2.0...v1.3.0

mgralikowski commented 3 months ago

Hi again, thx for improving my rules but unfortunately it works worse now.

  'files_count' => '{1} plik|[2,4] pliki|[5,*] plików',
  'person_count' => '{1} osoba|[2,4] osoby|[5,*] osób',
  'ball_count' => '{1} piłka|[2,4] piłki|[5,*] piłek',
  'coach_single' => '{1} trener|[2,4] trenerzy|[5,*] trenerów',

As you can see it adds pluralization even if it is not necessary.
Please remember that the most common pluralization is the version without an explicitly set number, just words separated by a pipe character, and we want to preserve it.
It stopped preserving letter capitalization.
Additionally, this rule you defined is wrong for Polish and probably for similar, local languages. It is not simply "2-4" - second form, and >=5 - third form, the rule is based on the last digit in a number, so it is more complex (yes, it's crazy to learn for foreigners...).

kargnas commented 3 months ago

I'm sad that the new code I improved performs even worse.

The primary reason I made the code somewhat complex is that I anticipated the need to consider it more thoroughly for other languages in the future.
I believe we cannot determine whether a string requires pluralization at this stage. Some English words maintain the same form in both singular and plural. Furthermore, the source language may not always be English. Some languages don't have pluralization at all. We need to account for these cases as well. To address these situations, all countable words should be incorporated into this pluralization system.
Concerning letter capitalization, I have implemented a fix in the master branch. I conducted tests using Claude 3.5.
Regarding the number system in Polish, I acknowledge your point. Many languages possess similar systems. However, we cannot accommodate this aspect since Laravel does not support localization based on the last digit.
Additionally, when translating countable words, it's more appropriate to include ":count" in the translation, such as ":count book|:count books" rather than just "book|books". This is particularly important as some languages have different word orders, and the placement of the count may vary.

mgralikowski commented 3 months ago

I'm sad that the new code I improved performs even worse.

Please don't get me wrong. I am not complaining, I want to make your solution better, so far it is the best library I tested. We want to implement auto-translation in our application.

I believe we cannot determine whether a string requires pluralization at this stage.

I think the best determinant is a simple condition - source translations contain a pipe character - which means we anticipate using this translation in a trans_choice() method. Even in the source language (EN in most cases) both forms are exactly the same, the developer should explicitly add both.

However, we cannot accommodate this aspect since Laravel does not support localization based on the last digit.

It does ;)

Additionally, when translating countable words, it's more appropriate to include ":count" in the translation

So far I haven't seen such a case, but I agree - It is more universal. Translations without variables are used by us in most cases for simple cases like charts, counters, and widgets.

kargnas commented 3 months ago

Please don't get me wrong. I am not complaining, I want to make your solution better, so far it is the best library I tested. We want to implement auto-translation in our application.

Don't worry, not that mean. I was just joking.

It does ;)

Oh, that's interesting.

So,

  'files_count' => '{1} plik|[2,4] pliki|[5,*] plików',
  'person_count' => '{1} osoba|[2,4] osoby|[5,*] osób',
  'ball_count' => '{1} piłka|[2,4] piłki|[5,*] piłek',
  'coach_single' => '{1} trener|[2,4] trenerzy|[5,*] trenerów',

Should that be like this, according to the Laravel translation system? Don't you need to specify the numbers?

  'files_count' => 'plik|pliki|plików',
  'person_count' => 'osoba|osoby|osób',
  'ball_count' => 'piłka|piłki|piłek',
  'coach_single' => 'trener|trenerzy|trenerów',

kargnas commented 3 months ago

I think the best determinant is a simple condition - source translations contain a pipe character - which means we anticipate using this translation in a trans_choice() method. Even in the source language (EN in most cases) both forms are exactly the same, the developer should explicitly add both.

For example, Korean don't use pluralization in sentences usually.

There are two pencils → 연필이 2개 있어요
There is one pencil → 연필이 1개 있어요

In the language file,

"pencil_exists" => "There is a pencil|There are :count pencils",

"pencil_exists" => "연필이 :count개 있어요",

연필 means pencil, and both cases are in the same form.

mgralikowski commented 3 months ago

Should that be like this, according to the Laravel translation system? Don't you need to specify the numbers?

Yes, are translations all valid.

I prepared more test translations:

[
    'years_passed' => '1 year passed|:count years passed', // an irregular noun in polish (rok->lata).
    'apple' => 'There is one apple|There are many apples', // it's not a precise number, so we need only 1 and 1+ forms.
    'orange' => '{0} There are no oranges|{1} There is one orange|[2,4] There are a few oranges|[5,*] There are many oranges', // custom ranges without a precise number
    'notifications_with_zero_value' => '{0} You don\'t have any notifications yet|[1,*]You have :count notifications', // not recommended way to define ranges (not applicable for some languages where based on the last digit)
    'notifications_without_zero' => 'You have :count notification|You have :count notifications', // recommended
    'notifications_empty' => 'You don\'t have any notifications yet', // a separated message when there are no notifications
    'person' => 'Person|People', // checking preserving letter capitalization
    'ball' => 'ball|balls', // a standard noun
    'trousers' => 'I have :count pants', // irregular (no singular) in EN and in PL as well
    'students' => 'student|students', // this is an exception - in PL there is only one plural version
]

I used Claude directly with a simple prompt:

Translate this Laravel translation array from English to Polish, including all grammatical aspects of Polish (exceptions, forms). If the English translation contains only 2 forms separated by a pipe and does not use ranges, add all the necessary forms in Polish (3 in most cases) also using a pipe character. If source translations use numbers or ranges, keep them in Polish the same. Don't add a :count placeholder if there is no such in the English translation.

[
    'years_passed' => 'Minął 1 rok|Minęły :count lata|Minęło :count lat',
    'apple' => 'Jest jedno jabłko|Są dwa jabłka|Jest wiele jabłek', // it is correct but the intention is to keep "one" and "few" options only
    'orange' => '{0} Nie ma pomarańczy|{1} Jest jedna pomarańcza|[2,4] Jest kilka pomarańczy|[5,*] Jest wiele pomarańczy',
    'notifications_with_zero_value' => '{0} Nie masz jeszcze żadnych powiadomień|[1,*]Masz :count powiadomień',
    'notifications_without_zero' => 'Masz :count powiadomienie|Masz :count powiadomienia|Masz :count powiadomień',
    'notifications_empty' => 'Nie masz jeszcze żadnych powiadomień',
    'person' => 'Osoba|Osoby|Osób',
    'ball' => 'piłka|piłki|piłek',
    'trousers' => 'Mam :count spodnie|Mam :count pary spodni|Mam :count par spodni',
    'students' => 'student|studenci|studentów'
]

Claude tends to add {2,4} so I had to explicitly forbid it. Almost all translations are correct!

Note - using numeric ranges, especially with 0 is tempting but it is not a recommended way to do so, because it makes declination rules not applicable in Polish and other languages. It is better to extract "no notifications" as a separate translation.

kargnas commented 3 months ago

@mgralikowski Can you look #3 and review the translations?

kargnas / laravel-ai-translator

Implement more advanced pluralization rules #1