Support for RTL input and display

[xsoh]

The editor doesn't recognize the non-latin characters (it writes nothing) even with changing the font of the editor. Also trying to paste a word of non-latin will print the Unicode representation (e.g. \u2320\u3433...etc) not the real word (e.g. سلام).

[reduz]

the default font does not, but you can set any custom font you like in the editor, that supports your character set.

https://github.com/okamstudio/godot/wiki/editor_font

On Wed, Dec 17, 2014 at 1:53 AM, Suhail Alkowaileet < notifications@github.com> wrote:

The editor doesn't recognize the non-latin characters (it writes nothing) even with changing the font of the editor. Also trying to paste a word of non-latin will print the Unicode representation (e.g. \u2320\u3433...etc) not the real word (e.g. سلام).

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982.

[xsoh]

Yes I have changed the font but no luck with that. The editor doesn't write anything and if paste it, it writes it as Unicode (decoded) chars.

[reduz]

oh when you change a font, you have to specify it to import unicode, otherwise it only imports latin. Still it's weird that it shows those characters, sounds like the text was not escaped (or more like, label will not unescape ext)

On Wed, Dec 17, 2014 at 1:12 PM, Suhail Alkowaileet < notifications@github.com> wrote:

Yes I have changed the font but no luck with that. The editor doesn't write anything and if paste it, it writes it as Unicode (decoded) chars.

[image: screenshot from 2014-12-17 18 53 12] https://cloud.githubusercontent.com/assets/871079/5474205/a39a8276-8620-11e4-8e78-56bf06bd9774.png

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-67345934.

[xsoh]

Oh sorry didn't see that. OK. Now the characters are shown but we have two challenges :

• the characters are written in LTR instead of RTL.
• the characters are not connected to each other (It should be like "سلام" not "س ل ا م"). Fribidi is a solution for that if its not using TrueType or Harfbuzz for TrueType.

Also still Ctrl + V didn't work for Unicode (E.g. سلام).

Thanks reduz.

[reduz]

this is opengl, so we can't use truetype. Do you know where documentation for supporting unicode RTL can be found?

On Wed, Dec 17, 2014 at 1:40 PM, Suhail Alkowaileet < notifications@github.com> wrote:

Oh sorry didn't see that. [image: screenshot from 2014-12-17 19 29 52] https://cloud.githubusercontent.com/assets/871079/5474710/79fc8884-8624-11e4-8eb9-1d259590212d.png OK. Now the characters are shown but we have two challenges :

• the characters are written in LTR instead of RTL.
• the characters are not connected to each other (It should be like "سلام" not "س ل ا م"). Fribidi https://github.com/behdad/fribidi is a solution for that if its not using TrueType or Harfbuzz https://github.com/behdad/harfbuzz for TrueType.

Also still Ctrl + V didn't work for Unicode (E.g. سلام).

Thanks reduz.

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-67350934.

[xsoh]

What about Fribidi? it doesn't need truetype. Its just apply the Bidi-Algorithim.

For that you can find it here Unicode Bidirectional Algorithm

[reduz]

fribidi is LGPL, can't be used on IOS, Android, consoles etc. will probably have to implement this myself.

On Wed, Dec 17, 2014 at 2:01 PM, Suhail Alkowaileet < notifications@github.com> wrote:

What about Fribidi? it doesn't need truetype. Its just apply the Bidi-Algorithim.

For that you can find it here Unicode Bidirectional Algorithm http://www.unicode.org/reports/tr9/

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-67354669.

[Masood-Lapeh]

Hi! [http://www.godotengine.org/forum/viewtopic.php?f=11&t=1232] I've the same problem with Farsi (Persian) characters... So I've to use some images instead... but It won't work everywhere... for example in text entry boxes... I'd appreciate it if you could fix it... Thanks for you attention!

[xsoh]

@Masood-Lapeh Yes, all the languages that written in the Arabic script language system will face same problem.

[reduz]

by the way, what do you guys use for text input? do you need a special IME popup or the keyboard just inserts the right characters directly?

[Evi1M4chine]

@reduz: Arabs have just normal keyboard layouts. The font rendering is a bit special though, as the characters combine into something that looks different. But I don’t think it matters, because I have an advanced German keyboard layout, and for all the unusual characters, when I press the key, Godot simply does nothing.

[reduz]

what platform are you on? maybe it has something to do with that, for me it works with english and spanish

On Fri, Jan 2, 2015 at 4:22 PM, Evi1M4chine notifications@github.com wrote:

@reduz https://github.com/reduz: Arabs have just normal keyboard layouts http://en.wikipedia.org/wiki/Arabic_keyboard. The font rendering is a bit special though, as the characters combine into something that looks different. But I don’t think it matters, because I have an advanced German keyboard layout http://www.neo-layout.org/, and for all the unusual characters, when I press the key, Godot simply does nothing.

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-68553803.

[Evi1M4chine]

@reduz: Might be. This is

• amd64 platform,
• Linux 3.18.1,
• Xorg 7.4 (server 1.16.2.901, but with nVidia drivers 340.65),
• KDE 4.14.3,
• GCC 4.8.3 (LLVM 3.5.0 is installed too)
• Python 2.7.8 (3.3.5 is installed too, but disabled via eselect, as Godot doesn’t work with it)
• SCons 2.3.4
• Mesa 10.3.5
• ALSA libs 1.0.28
• Freetype 2.5.4
• OpenSSL 1.0.1j
• pkgconfig 0.28
• and Godot is obviously self-compiled from the latest git sources.

I just tested it a bit (trying to enter special characters in a label text field and found at least one character for each mod key combination that worked. It seems to be a case of a limited character set. It also apparently seems to slow down, because the invisible characters still count. So when using the cursor or pressing backspace, it may look like nothing happened, or as if it was hanging. This was really hard to find out. As for rendering them in the scene: I could not find a character that showed up in the text field but not in the scene. Probably because I imported and used my own full Unicode font. (I know that font works, because I use it for the rest of the system.)

I could not test advanced layout things though (like Arab or Thai scripts)…

Anyway, this is not that important to me. I’m just interested in having GDscript identifiers with Unicode characters in them, and the basic German letters. Everything special can be done by making my own images. :)

[godotengine]

question, what do you do with the NEO keyboard layout with shortcuts such as ctrl-z, ctrl-s, etc?

On Fri, Jan 2, 2015 at 7:02 PM, Evi1M4chine notifications@github.com wrote:

@reduz https://github.com/reduz: Might be. This is

• amd64 platform,
• Linux 3.18.1,
• Xorg 7.4 (server 1.16.2.901, but with nVidia drivers 340.65),
• KDE 4.14.3,
• GCC 4.8.3 (LLVM 3.5.0 is installed too)
• Python 2.7.8 (3.3.5 is installed too, but disabled via eselect, as Godot doesn’t work with it)
• SCons 2.3.4
• Mesa 10.3.5
• ALSA libs 1.0.28
• Freetype 2.5.4
• OpenSSL 1.0.1j
• pkgconfig 0.28
• and Godot is obviously self-compiled from the latest git sources.

I just tested it a bit (trying to enter special characters in a label text field and found at least one character for each mod key combination that worked. It seems to be a case of a limited character set. It also apparently seems to slow down, because the invisible characters still count. So when using the cursor or pressing backspace, it may look like nothing happened, or as if it was hanging. This was really hard to find out. As for rendering them in the scene: I could not find a character that showed up in the text field but not in the scene. Probably because I imported and used my own full Unicode font. (I know that font works, because I use it for the rest of the system.)

I could not test advanced layout things though (like Arab or Thai scripts)…

Anyway, this is not that important to me. I’m just interested in having GDscript identifiers with Unicode characters in them, and the basic German letters. Everything special can be done by making my own images. :)

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-68566711.

OkamStudio

[Evi1M4chine]

Well, the Ctrl, Alt and Win keys are still available. Maybe the fact that in Germany the Ctrl key is called “Strg” was confusing.

So I just press them like normal.

[Just that of course I have more other combinations available, as I could set shortcuts to Ctrl-Alt-Mod3-Mod4-(someletter) if I was crazy. ;) Even KDE, and Gnome sometimes fail to handle those correctly though. E.g. because Mod3-A is actually {, and so Shift-{ is actually Mod3-Shift-A… but that is also the letter α, and so is ambiguous.]

[mohaalak]

@reduz I contacted Behdad programmer of Fribidi for licensing issue on Fribidl , I am waiting for an answer.

[reduz]

That looks o,k, though I don't really understand how it's supposed to work. As far as I understand, the font engine must support the following? 1) RTL and LTR characters, for switching direction? (as far as I understand it's like characters are pushed, then retrieved in the opposite direction?) 2) Ligatures between different types of character combinations

Is there anything else?

On Mon, Mar 9, 2015 at 9:05 AM, Mohammad Hadi Aliakbar < notifications@github.com> wrote:

@reduz https://github.com/reduz I contacted for licensing issue on Fribidl , I am waiting for an answer.

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-77841714.

[mohaalak]

1 - in RTL langauges all character go from left to right but digits and LTR characters should go from left to right (it's not like align left) 2- yeah correct the combination of characters should print something else.

you think we can create from ground up this feature without using any library?

if you point me in right direction I think I can make it work at least for Farsi and Arabic.

should I change https://github.com/okamstudio/godot/blob/master/scene/resources/font.cpp

draw function?

.

[reduz]

Ok, but as far as I understand Unicode by default is LTR, so if unicode supports RTL it must have some sort of character that tells you "ok from now on, following text is RTL" and then something to also tell "ok from now on, following text is LTR again".

On Mon, Mar 9, 2015 at 11:07 AM, Mohammad Hadi Aliakbar < notifications@github.com> wrote:

1 - in RTL langauges all character go from left to right but digits and LTR characters should go from left to right (it's not like align left) 2- yeah correct the combination of characters should print something else.

you think we can create from ground up this feature without using any library?

if you point me in right direction I think I can make it work at least for Farsi and Arabic.

should I change https://github.com/okamstudio/godot/blob/master/scene/resources/font.cpp

draw function?

.

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-77859163.

[mohaalak]

every text editor that support RTL has a button to make text RTL or LTR. we should add a property that make control is on RTL or LTR.

[reduz]

oh so this is external to the text.. you mean that label should have a button that says "Use RTL Text"

However, if we were dealing with game translations and we wished to support RTL languages, how is this handled?

On Mon, Mar 9, 2015 at 3:51 PM, Mohammad Hadi Aliakbar < notifications@github.com> wrote:

every text editor that support RTL has a button to make text RTL or LTR. we should add a property that make control is on RTL or LTR.

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-77917673.

[xsoh]

This project might be good for understanding Bidi https://github.com/salshaaban/BidiRenderer

[mohaalak]

it's a little tricky , when I want to use it on web , after translation we should set direction to RTL , here we should have a system that if this language is RTL or not , if it is RTL the Controller should change from LTR to RTL and every object that inherit this should become RTL.

in some softwares when you change your layout from LTR to RTL everything change from left to right, like when in a form, labels is left and inputs are in front , in right of screen , when change to RTL mode in some softwares it change all things , labels goest to right of screen, inputs go to left of screen .

It's living in hell to write multi-language software when RTL language is concerned :D

I think if labels and text edits can support RTL is fine.

@xsoh this code comes from a library called Unicode bidirectional algorithm , and this algorithm is GPL. I don't know much about licensing but @reduz say we can't use it cause in android , ios , consoles you can't use GPL

[lostdj]

this is opengl, so we can't use truetype.

Yes you can. :)

[xsoh]

@mohaalak Does the algorithm it self (I don't mean the code. I mean the algorithim) has licensing? What I meant for BidiRenderer is just for understanding not using the code it self.

[mohaalak]

I read in fribidI that algorithm has license too. I don't get these licensing issues and after a little research Fribidi might not be our best option . There is another library named harbuzz it use true type and it has 'old MIT' License with Cairo library we could support true type fonts but I think it's too much work to change how fonts draws now and there will be lots of bugs and compatibility issue with older version of godot. after reading source code of harfbuzz I think I could change it to use BMfonts .I think it gets only size of each glyph from ttf . I could pass size from BMfonts to harfbuzz . it should be interesting. On Mar 14, 2015 11:21 AM, "Suhail Alkowaileet" notifications@github.com wrote:

@mohaalak https://github.com/mohaalak Does the algorithm it self (I don't mean the code. I mean the algorithim) has licensing? What I meant for BidiRenderer is just for understanding not using the code it self.

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-80097574.

[mohaalak]

I have good news and bad news.

Good News :

• implement RTL support for Godot in this gui interfaces:
  • label
  • text edit
  • line edit
  • button
• another good news is that if you write RTL language , it will automatically change to RTL mode. @reduz so I think in translation everything is safe

Bad News :

I could not work with harfbuzz , it needs true type support and for that we should use cairo library and font is gonna be change fundamentally. I think @reduz and other main developers should discuss about this change (I could do it but first let's discuss it ) , a post about cairo and bmfonts

I used Fribidi ,it's license is LGPL, it means that you can't release it to Android , iOS or anywhere else unless , users can change just Fribidi library ,link it to your code and everything else works again. it can be done.

1. you can open source your game ( not a great idea for everyone)
2. you can export a release mode from godot and let's anyone that wants, compile godot from repository with their changes and use your released datapack( I don't know if it's doable or not)
3. you can change scons to build a Static Library for Fribidi and ship your game with this library I know it works in android , windows , linux but for ios I just found this link, short answer is that you should ask permission of Fribidi to use it under iOS, I don't know anything about other consoles.

you can clone my repository and work with it if you want , I don't open a pull request I don't think @reduz wants this changes into main repository.

[mohaalak]

by the way we can use BIDI algorithm but I don't know much about licensing , it does not mention anything here about any license but I think I read somewhere in fribidi doc that algorithm is LGPL also. I found this source codes for bidi algorithm , I will work with this sample code too , maybe I can make something out , but if we use truetype it's gonna be better.

[mohaalak]

above algorithm work for RTL support but we need Ligatures for other languages. I can make something for arabic and persian but I don't know about other languages.

[mohaalak]

ok I found something for reshaping ftp://ftp.unicode.org/Public/UNIDATA/ArabicShaping.txt yess!!! I think I can make this work without any licensing issues. I will update in a few days.

[godotengine]

Algorithm should work fine, no licensing issues unless it's a library. For ligatures, maybe freetype handles this as special cases of character combinations?

On Sat, Mar 21, 2015 at 7:51 PM, Mohammad Hadi Aliakbar < notifications@github.com> wrote:

ok I found something for reshaping here. yess!!! I think I can make this work without any licensing issues. I will update in a few days.

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-84466190.

OkamStudio

[mohaalak]

@okamstudio unfortunately freetype does not handle this but unicode.org shared a text that list all joining types in Arabic, Syriac, N'Ko, Mandaic, and Manichaean , Mongolian, Phags-pa, and Psalter Pahlavi , I think I can work something out with this info.

[reduz]

wait, so basically unicode provides all shaping types as different characters? so you need to basically depending on the character and next character use a different character code?

On Sat, Mar 21, 2015 at 8:12 PM, Mohammad Hadi Aliakbar < notifications@github.com> wrote:

@okamstudio https://github.com/okamstudio unfortunately freetype does not handle this but unicode.org shared a text https://github.com/roozbehp/unicode-data/blob/master/ArabicShaping.txt that list all joining types in Arabic, Syriac, N'Ko, Mandaic, and Manichaean , Mongolian, Phags-pa, and Psalter Pahlavi , I think I can work something out with this info.

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-84469669.

[mohaalak]

yeah , I should check next and previous character to decide what character code I should use. look at this ت‎ ـت‎ ـتـ‎ تـ‎ all of this letters above is spelled T in arabic first one isolated, second one is joined from left , third joined from both side , forth one joined from left

look at this

but there is more :smiley: sometimes two characters combine and produce one character this way your string size will change. س ل ا م سـ لا م

there should be a map for all of this combinations.

[reduz]

from what you say, all points to the fact that there should be a method in String, something like: "String normalize(bool start_rtl) const" or probably that goes through the string and replaces characters and combinations to make them correct, so it works properly with draw_text()...

we could try implementing this first, then see how to use this method in Label, Button and other controls

On Sat, Mar 21, 2015 at 8:50 PM, Mohammad Hadi Aliakbar < notifications@github.com> wrote:

yeah , I should check next and previous character to decide what character code I should use. look at this ت‎ ـت‎ ـتـ‎ تـ‎ all of this letters above is spelled T in arabic first one isolated, second one is joined from left , third joined from both side , forth one joined from left

look at this http://en.wikipedia.org/wiki/Template:Arabic_alphabet_shapes/joining

but there is more [image: :smiley:] sometimes two characters combine and produce one character this way your string size will change. س ل ا م سـ لا م

there should be a map for all of this combinations.

— Reply to this email directly or view it on GitHub https://github.com/okamstudio/godot/issues/982#issuecomment-84474906.

[mohaalak]

yeah we need that function in String , in my implementation of fribidi in Godot I created a class with a static function , but you are right it should be in String. I think we should add another string for label , button and other control , a string variable that what client can see , and an original text.

1. you want to call normalize just one time when set_text called
2. when a text is normalized if you call normalize again it generates some nonesense ( that was the case in Fribidi , I don't know maybe it was a bug from Fribidi)
3. sometimes the length of strings will change , when user remove one character you should have remove from originalText then update VisualText
4. when get_text called you should return the original text

look at my changes in line_edit (sorry for bad indenting)

also in _notifiacation, drop_selected , _input_event.

[lostdj]

I have no idea how complete the support is https://github.com/mosra/magnum-plugins/tree/master/src/MagnumPlugins/HarfBuzzFont And it still will require FT support.

[lostdj]

@okamstudio

no licensing issues unless it's a library

Except there are. FriBidi used by @mohaalak is GPL. IANAL, but it's risky to depend on it, even if your game depends on a library with permissive license (Godot) and is opensource, which depends on a library with copyleft license. And that's not counting the issues with *GPL licenses vs. Apple.

[mohaalak]

@lostdj harfbuzz needs freetype fonts and Godot does not support freetype, it uses ttf to make a bitmap font and use it afterwards. Fribidi uses LGPL but yes our problem is license issues and I am working to code from ground up without needing any GPL or LGPL library.

[behdad]

/sub

[akien-mga]

Renamed the issue for clarity as most of the discussion revolves around RTL input.

[reduz]

there is PR for this that works pretty well, but depends on a large external library for a single function. The function should be just be extracted or rewritten

On Sun, Nov 1, 2015 at 9:07 AM, Rémi Verschelde notifications@github.com wrote:

Renamed the issue for clarity as most of the discussion revolves around RTL input.

— Reply to this email directly or view it on GitHub https://github.com/godotengine/godot/issues/982#issuecomment-152821966.

[behdad]

there is PR for this that works pretty well, but depends on a large external library for a single function. The function should be just be extracted or rewritten

Note that that single function is exercises most of the library, if FriBidi is the library you are talking about.

If licensing is a problem, I was going to convince you to rewrite based on HarfBuzz, but if you call FriBidi "large", then HarfBuzz is HUGE!

[reduz]

It seems Unicode has a simple source code reference of he bidirectional algorithm: http://www.unicode.org/Public/PROGRAMS/BidiReferenceCpp/v25g/ anyone that understands enough could adapt it?

[akien-mga]

It seems Unicode has a simple source code reference of he bidirectional algorithm: http://www.unicode.org/Public/PROGRAMS/BidiReferenceCpp/v25g/ anyone that understands enough could adapt it?

Actually there's a version 26 of that algorithm too: http://www.unicode.org/Public/PROGRAMS/BidiReferenceCpp/v26/

To anyone interested in tackling this, please check #2886 as a reference for a WIP implementation using the ICU library. We believe that we'd be better off without adding a big library like ICU just for that, hence the proposal to use the Unicode Bidi reference, but at the same time most of us are not RTL experts and might miss something.

[mohaalak]

it's 1 year that I don't make any games with godot, but I hope for Version 3.0 it will support RTL, I don't have time to implement it but I can test it if you want, please make this happen

[reduz]

@mohaalak Gandhi once said "Be the change you want to see in the world". If we don't get help from someone who understands how RTL is used, we simply can't do it.

[khaledhosny]

Proper Arabic/Indic/etc and RTL support will require a punch of extra dependencies as it is a rather huge amount of work to do from scratch or fork and bundle existing libraries. Are such dependencies something the maintainers are open to?

[akien-mga]

@khaledhosny Up to now the general feeling is that the less additional dependencies, the better. We were under the impression that it should be possible to reimplement the BiDi algorithm directly (or via a lightweight library) - but actually the more time passes, the more I think there might be a reason why RTL libraries are so big (especially as Arabic letters need to support mutating based on their context).

So if a RTL-aware dev could do some research about what would be a minimal set of dependencies/features needed to support RTL and text shaping properly on all platforms, that would help bringing this issue closer to a fix.

[khaledhosny]

Minimal dependencies would be FriBiDi for bidirectional support, HarfBuzz for font shaping, FreeType for font rasterization (not a hard requirement) and some code to glue everything together (Raqm can be that glue but it is not absolutely needed). If FriBiDi license is an issue then ICU is another replacement, but otherwise I recommend against just using the sample code from Unicode unless someone has the time an interest in actively maintaining it in this code base.

This should give you support for Arabic, Hebrew, lots of Indic and South Asian scripts and more.