Add basic Unicode support

MadCatX commented 8 months ago

This patch adds basic Unicode support to the UI. It relies on Freetype font renderer and is hidden by default behind UI_UNICODE flag. Encoding support is currently limited to UTF-8.

What works:

Display of all Unicode characters. Tested against (https://github.com/bits/UTF-8-Unicode-Test-Documents/blob/master/UTF-8_sequence_separated/utf8_sequence_0-0x10ffff_assigned_including-unprintable-asis.txt) and (https://github.com/bits/UTF-8-Unicode-Test-Documents/blob/master/UTF-8_sequence_separated/utf8_sequence_0-0x10ffff_assigned_including-unprintable-replaced.txt)
Handling of invalid UTF-8 input. Invalid characters are replaced with ? without any other adverse effects.
Navigation in text using keyboard and mouse, selection, copy-paste, replacements.
Starting executables from paths with Unicode characters. The path currently needs to be pasted into the textbox.

What could be improved:

Word-skipping in text with Unicode characters is a bit flaky. To fix this, I suppose that the IsCharAlpha() function would have to be much more sophisticated.
Cursor position selection with mouse and invalid UTF-8 input. Sometimes the cursor is positioned a few characters behind the expected position if the expected position lies inside an invalid UTF-8 sequence. Navigation with keyboard seems to work fine.

What does not work:

Textboxes currently do not accept Unicode input from keyboard keystrokes. A workaround is to copy-paste it in.

Notes:

This patch changes the meaning of carets in UIStringSelection to by how many glyphs the caret is offset from the beginning rather than bytes.
This patch also fixes a minor rendering bug when a text selection in UITextbox contains tabs.

Comments welcome :)

nakst commented 8 months ago

Thanks for making this, I'll review it properly when I have time (I'm a bit busy at the moment).

MadCatX commented 7 months ago

Thanks for reviewing the PR. Don't worry about the coding style change requests, it's your code and your rules. I'll see what I can do in the upcoming days as I'm a bit busy myself ATM.

MadCatX commented 6 months ago

My apologies for taking so long to get back to this. I addressed the coding style issues and factored out some parts of functions to macros to avoid the convoluted ifdefs.

I still need to investigate the cursor positioning issue. It appears that I can repro it even with unicode handling switched off. Hopefully I should have more time to work on this now...

MadCatX commented 6 months ago

I think I figured out the cursor placement issue. Let me know if you'd like me to revise the code further or do some additional tests.

nakst commented 6 months ago

Thanks, I'll try to take a look at the weekend.

MadCatX commented 6 months ago

Thanks for the comments. I've generalized cursor positioning and byte<->column translation functions. Hopefully I've also fixed an issue where the code would assert on an attempt to get a code point from an empty string. As far as I could test things seem to behave correctly.

MadCatX commented 4 months ago

I'm sorry for the ping... is there anything else you'd like me to revise?

nakst commented 4 months ago

Sorry, I have it on my list of things to-do to review these changes. But I have been busy/tired lately, so I haven't done it yet.

nakst commented 2 months ago

Thanks for submitting these changes!

nakst / gf

Add basic Unicode support #169