Cursor jumps between start and end of line when line begins with a RTL letter while typing with non-RTL letters and symbols

[dhoverml]

Tested versions

v4.3.dev5.official [89f70e98d]

System information

Apple M1, Ubuntu 20.04 LTS

Issue description

See title.

Steps to reproduce

1. Optionally download the MRP (rtl_jumps.zip) then go to step 9. Otherwise follow steps 2-8 to recreate the MRP.

2. Create a csv called translations.csv and add the following:

   key,en,ar test,test,test

3. Create a new project.

4. Click on User Inteface and replace the root node with a LineEdit node.

5. Move translations.csv created in step 2 into the new project.

6. Project -> Project Settings...

7. Click on the Localization tab, click Add... and choose translations.ar.translation and translations.en.translation.

8. Click on the General tab, toggle Advanced Settings, Internationalization -> Locale, set Test to ar.

9. Run the project.

10. Paste this Arabic letter as the first letter: ﺵ

11. Alternate between English letters and symbols, like a,b<c>d;e:f, and watch the cursor jump back and forth.

https://github.com/godotengine/godot/assets/135338122/31032bb4-0e11-49dd-beca-7087bdc3a9bd

Notes

1. ( ), { }, [ ], are confusing since the opposite brace appears, though I don't know what the expected behavior should be.
2. The issue does not occur if an English letter was typed first. However, the cursor behaves strangely once an Arabic letter is typed. For example, type this: abc then paste ﺵ. Notice the cursor is to the left of ﺵ. If you type another symbol or English letter the cursor will jump to the right. Again, I don't know if this is expected.
3. You can see https://github.com/godotengine/godot/issues/84370 in the video

Minimal reproduction project (MRP)

rtl_jumps.zip

[bruvzg]

( ), { }, [ ], are confusing since the opposite brace appears, though I don't know what the expected behavior should be.

This and neutral characters jumping is expected (all other UI toolkits I have tested act exactly the same way). Since the first character is Arabic, the base direction of the string is RTL and neutral characters are considered to be RTL as long as they aren't part of RTL run.

Cursor jumping is also expected (other UI toolkits do the same). Note: this is not how cursor in the native macOS text fields works, but macOS is an exception.

[bruvzg]

Qt 6.5.1 for example:

https://github.com/godotengine/godot/assets/7645683/a8bbebb8-8a3b-48ca-bc09-23daff62093d

[bruvzg]

But I guess we can add an option to use macOS style "visual" caret movement, in addition to the current one (won't be easy to do, due to the way selection works).

[dhoverml]

After comparing Godot with other applications when mixing LTR and RTL (bidi?), I think this bug can be closed.

However I'm leaving it open since I think some documentation should be added (to https://docs.godotengine.org/en/stable/tutorials/i18n/internationalizing_games.html) to explain:

1. Why the cursor jumps while typing (and that it is normal and seen in other applications)
2. Why symbols can teleport around (symbols are treated as RTL unless if they're followed by LTR, or viceversa, depending on context)
3. Why pressing the arrow keys doesn't always move the cursor in its direction (it depends on the text it is over)
4. What the angle brackets mean (seen when you move the cursor between LTR and RTL sections).

Even though this behavior (except 3 and 4) can be seen in other applications, I think it could be helpful since it can differ enough to warrant an explanation.

Also, not this bug exactly, but I still have trouble navigating a LineEdit with the arrow keys when it mixes LTR and RTL. The cursor changes direction depending on where it is. I understand why it does it, but it isn't always obvious as a user. Other applications (Browser URL bar, Ubuntu Settings search, GEdit) did not exhibit this behavior: when you press left-arrow, the cursor moves left. When you press right arrow, the cursor moves right. Maybe this is what @bruvzg meant by "visual" caret movement?

[bruvzg]

Also, not this bug exactly, but I still have trouble navigating a LineEdit with the arrow keys when it mixes LTR and RTL. The cursor changes direction depending on where it is.

This is also expected and used by a lot (probably most) UI toolkits, the cursor moves the same way text is stored, pressing left moves it to the next character (so if it's on RTL portion of the text it will move right).

when you press left-arrow, the cursor moves left.

That's what I meant by "visual" (never saw this type of behavior outside of macOS).

[bruvzg]

1. Normal caret (full height vertical line) - used in most cases, when caret position and direction are precisely determined.
2. Grapheme caret (full height vertical line with a bar on the top, T shape) - used when caret is in the middle of a grapheme cluster (e.g., ligature) and its position is not precisely determined.
3. Split caret (two half height lines, with additional horizontal line) - used on the border of text parts with different writing direction. Bottom half is drawn after the previous character, top half before the next character, horizontal part indicate writing direction of the previous and next character respectively.

There's no standard how to show it (I have seen arrows, vertical splits, color and similar "brackets", or nothing in the different OSes/toolkits), so documenting it would make sense.