Chinese characters do not render correctly

[ThexXTURBOXx]

Description Chinese characters render as question marks.

Steps to Reproduce Specific steps to reproduce the behavior:

1. Let RSTA render chinese text in some way (in my case, loading it from a file)
2. The characters render as unknown characters (similar to �)

Expected behavior The chinese characters should be shown as usual

Actual behavior The characters render as unknown characters (similar to �)

Screenshots See issue at my repo: https://github.com/ThexXTURBOXx/Luyten/issues/2

Java version Tested myself on Java 8 and Java 17

Additional context

Chinese characters were working fine in 3.2.0, but they do not work in 3.3.0 (and newer). I have searched within the changes between the two versions: https://github.com/bobbylight/RSyntaxTextArea/compare/3.2.0...3.3.0 And I suspect that adding these lines is the culprit, however I am not entirely sure and have not tested this properly.

This issue was reported to me via https://github.com/ThexXTURBOXx/Luyten/issues/2. If you need some chinese text to test this on your end, here you go: Test.txt

[bobbylight]

Are you using a custom Theme? How is the RSTA initialized as far as colors and fonts are concerned?

[ThexXTURBOXx]

Luyten only uses built-in Themes, loaded via their respective XML file, as seen here. Actually applying the Theme and Font happens here. It is worth mentioning that one can also set the Font during runtime, as seen here. However, I have not yet tested re-setting the font when Chinese text is currently being displayed.

[Mgazul]

After testing, the problem is related to Theme. You need to switch the idea theme once before Chinese can be displayed correctly.

[bobbylight]

@ThexXTURBOXx @Mgazul there have been a couple of changes in font management. As long as the primary font of the text area, i.e. that set by RSyntaxTextArea.setFont(), supports Chinese, it should render properly for all token types (unless you use multiple different fonts in your custom Theme or SyntaxScheme, in which case each one would need to support Chinese). Further, if you use Font.MONOSPACED, the JVM should automatically use a fallback font for code points not supported by the current font.

Note that I believe RSTA's Windows default monospaced font at least, i.e. Cascadia Code unfortunately does not support CJK characters. We pick that because it's what many Windows apps now use. I believe Font.MONOSPACED defaults to something like Courier New, which isn't as nice but should fallback and support all code points.

Anyway give the master branch a shot if you can and let me know if you see a difference.

[ThexXTURBOXx]

@bobbylight Thank you very much for your answer! Indeed, after trying out the latest master commit, it worked "better" on my end. It was not working perfect, though. So, I investigated a bit more. For example, when scrolling up and down, Luyten changes the font size, and it does this using

textArea.setFont(new Font(textArea.getFont().getName(), textArea.getFont().getStyle(),
        luytenPrefs.getFontSize()));

As you can see, this is creating a new Font instance - and something within RSTA does not like that. I changed all these to simply

textArea.setFont(textArea.getFont().deriveFont((float) luytenPrefs.getFontSize()));

and suddenly zooming was working fine! So, apparently it is important to use Font#deriveFont instead of creating a new Font instance, because the latter probably does not retain some internal state.

Anyway, TL;DR: Everything works fine on my end using the latest master now! Thank you very much and I will be patiently waiting for the next official release! I will keep this issue open for now - feel free to close it at any point (I am unsure whether you want to close it now or after the release of the next version!)

[bobbylight]

Yes, you have to use either Font#deriveFont(), to preserve all attributes of the oringal font including fallback behavior, or StyleContext.getDefaultStyleContext().getFont(...) which should lookup and use the fallbck behavior for the Font based on some internal Swing cache. This stuff is a little finnicky, as I saw recently when I was working on the ligature support just released. You have to instantiate your Fonts a certain way for certain features but I'm glad things are working for you!

I'll close this ticket out, but please open a new one if you find some cocktail of Theme or SyntaxScheme usage that seems to "break" CJK rendering that worked before.