Pasting "〽️" in TextArea Moves Cursor Too Far

[jbdyn]

Hi! :wave:

Problem

After pasting 〽️ in TextArea, the selection is moved too far and I need to press Backspace twice (instead of once) to delete it. Probably other emojis also cause this, but I have not tested this.

I tried to dive into the code, but unfortunately could not find where to fix this.

Example

Starting with a blank TextArea, the selection is given with (start, end) == ((0, 0), (0, 0)). Inserting a would give (start, end) == ((0, 1), (0, 1)), as expected. Inserting 🌴 ("palm tree") would also give (start, end) == ((0, 1), (0, 1)). Nice. However, pasting 〽️ ("part alternation mark") sets the selection to (start, end) == ((0, 2), (0, 2)).

Test Script

# Test Script ```python from textual.app import App from textual.widgets import TextArea class TestApp(App): def compose(self): yield TextArea() def main(): app = TestApp() app.run() if __name__ == "__main__": main() ```

Textual Diagnostics

# Textual Diagnostics ## Versions | Name | Value | |---------|--------| | Textual | 0.71.0 | | Rich | 13.7.1 | ## Python | Name | Value | |----------------|--------------------------------------------------------------------------------| | Version | 3.12.4 | | Implementation | CPython | | Compiler | GCC 14.1.1 20240522 | | Executable | /home/jbdyn/.cache/pypoetry/virtualenvs/textual-VZw8wrTS-py3.12/bin/python | ## Operating System | Name | Value | |---------|-----------------------------------------------------| | System | Linux | | Release | 6.9.3-3-MANJARO | | Version | #1 SMP PREEMPT_DYNAMIC Mon Jun 10 09:50:04 UTC 2024 | ## Terminal | Name | Value | |----------------------|-----------| | Terminal Application | *Unknown* | | TERM | foot | | COLORTERM | truecolor | | FORCE_COLOR | *Not set* | | NO_COLOR | *Not set* | ## Rich Console options | Name | Value | |----------------|----------------------| | size | width=171, height=83 | | legacy_windows | False | | min_width | 1 | | max_width | 171 | | is_terminal | False | | encoding | utf-8 | | max_height | 83 | | justify | None | | overflow | None | | no_wrap | False | | highlight | None | | markup | None | | height | None |

[github-actions[bot]]

Thank you for your issue. Give us a little time to review it.

PS. You might want to check the FAQ if you haven't done so already.

This is an automated reply, generated by FAQtory

[TomJGooding]

Just to test a theory, could you try pasting this version of the emoji instead?

〽

This was generated with print("\u303d")

[jbdyn]

This works: Pasting 〽 ("\u303d") gives (start, end) == ((0, 1), (0, 1)).

[darrenburns]

When you paste 〽️, you're pasting 2 codepoints into the TextArea. When you press backspace after this, you're only deleting the second codepoint.

The 🏴󠁧󠁢󠁳󠁣󠁴󠁿 emoji, for example, contains 7 codepoints, so you'd need backspace 7 times to delete them all.

This aligns with how VSCode and PyCharm behave too.

Python 3.11.8 (main, Apr 25 2024, 11:02:29) [Clang 13.0.0 (clang-1300.0.27.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> len("🏴󠁧󠁢󠁳󠁣󠁴󠁿")
7
>>> len("〽️")
2
>>> len("🌴")
1

[jbdyn]

This makes sense to me now, thank you.

Indeed, also Vim and Neovim insert seven columns with 🏴󠁧󠁢󠁳󠁣󠁴󠁿. In contrast, the terminal foot and messengers Signal and in-browser Element only require the user to press Backspace once for one 🏴󠁧󠁢󠁳󠁣󠁴󠁿. Although I know now better, the skew between inserted columns and displayed content is still distracting to me in all occurrences while typing, to be honest. I am used to emoji behaving as single characters, which should be the default in user interfaces anyway, shouldn't it?

What do you think? Will there be a way for TextArea adapting to a more terminal-/messenger-like user experience when it comes to typing emoji?

[darrenburns]

Do you mean you're seeing some kind of visual glitch too? I'm unsure if that's what you mean by "skew". Could you post a screenshot if so?

[jbdyn]

With "skew" I just mean the difference between the insertion of 1 emoji and the resulting displacement of the cursor by potentially more than 1 cell, in any editor. Sorry for being unclear on that.

In the case of TextArea, there is no obvious visual glitch and the cursor moves - visually - as expected. However, it feels glitchy for me moving backwards with Backspace over visually hidden code points while getting no visual feedback (the cursor stays for some key presses in the case of 🏴󠁧󠁢󠁳󠁣󠁴󠁿).

I mean, this is a minor issue after all. It's a bit weird to me having to press Backspace seven times to delete one flag emoji, but technically the behaviour is consistent and solid. I can imagine that getting TextArea to see 🏴󠁧󠁢󠁳󠁣󠁴󠁿 as any other single character is not an easy task and might involve parsing, chunk maps or something else.

What is your view on that?

[TomJGooding]

I think you've already summed it up nicely:

• This is a minor issue
• Fixing this would not be an easy task
• Current behaviour is consistent with other editors

If Microsoft haven't figured out how to treat emoji sequences as single clusters in VSCode, perhaps this is a bit much to expect from Textual?

[jbdyn]

[...], perhaps this is a bit much to expect from Textual?

I have high hopes :relaxed:

I won't close this issue myself because I would really like to see this feature implemented, but I am also happy with you closing this as "won't implement". Feel free :slightly_smiling_face:

Thank you both for your quick replies, @TomJGooding and @darrenburns. Always nice to see. :1st_place_medal:

[darrenburns]

To do this we'd probably need to introduce a new dependency for segmenting text into grapheme clusters, e.g. https://uniseg-py.readthedocs.io/en/latest/graphemecluster.html. As mentioned though, you're unlikely to see this any time soon inside Textual itself as it's pretty low priority.

If you really want this, it might not be too difficult to do it in a subclass of TextArea (and overwriting the "cursor left/right" and "delete left/right" actions to make use of the library linked above (specifically this).

I don't have the bandwidth to look into the exact code that would be required, but I'm happy to give a bit of guidance/pointers if you try it in a subclass.

[jbdyn]

Cool, thanks :heart: Then I will take a look at it. :+1: