multibyte languages rendering is strange(9.0.472)

[shikato]

Steps to reproduce

1. Launch MacVim 9.0.472
2. Input multibyte languages
   • e.g.
     • こんにちは
     • 안녕하세요
3. The text rendering is so strange.

Expected behaviour

I expect normal rendering.

Version of Vim and architecture

9.0.472

Environment

macOS 12.3.1 (21E258)

How MacVim was installed

No response

Logs and stack traces

No response

Vim configuration where issue is reproducable

No response

Issue has been tested with given configuration

• [ ] by running MacVim.app from GUI macOS interface
• [ ] by running vim/gvim/etc installed by MacVim
• [ ] by running other versions of vim (e.g. /usr/bin/vim)

Issue has been tested with no configuration

• [ ] by running mvim --clean (or gvim, supplied by MacVim distribution)
• [ ] by running vim --clean (in terminal, supplied by MacVim distribution)
• [ ] by running vim --clean (in terminal, other suppliers, e.g. /usr/bin/vim)

Other conditions

• [ ] The both Homebrew packages "vim" and "macvim" are installed

[shikato]

I checked that the bug do not occur in version 9.0.0065. https://github.com/macvim-dev/macvim/releases/tag/snapshot-173

[eirnym]

I have a test for various characters in my vimrc (demonstration you see on a screenshot below). My guess it is connected with drawing emoji and other wide characters from fallback font in general.

As far as I remember (as it was discussed before at least), the rectangle was calculated from the main font and to other fonts were cut as they were wider. newer implementation just doesn't cut the character size from fallback font and you see a mix. Long story short, you can control if main font is monospaced, but you can't control spacing from another fonts. Also you can't check and control all characters in a whole document you edit as it could be way bigger than you observe. (I advocate here to Vim's implementation, where what you see on a screen is what Vim renders, and it renders a very small portion of it. Because of clever implementation, Vim can easily edit very huge files without a penalty)

I personally prefer this implementation than a lot of font and document hacks.

In screenshot below, rightmost right arrow on a line above emoji takes the same width as w above as these symbols are taken from Fira Code Light I use. On the other hand, some non-English, Japanese, "Powerline" and Emoji characters are taken from fallback system fonts, so could have their own width (which is not exactly the same as English letters). You can see this discrepancy in "Powerline" section: lines are looks different despite they have exactly the same amount of characters in them. Similar situation you'll see in emoji: first emoji is in the same column as arrow I mentioned above, but way wider.

Thanks to a good implementation, movement works "visually" correctly: despite every japanese hieroglyph is a single character, when I move up from the second, my cursor will move to ι, not to ε.

[eirnym]

I described how I remember development of MacVim and my personal opinion. It could be different from yours as I don't type often in other languages than European. I even don't have special fonts for eastern languages

[ychin]

I can't really reproduce this. Some questions for you @shikato:

1. Have you run this with mvim --clean as prompted by the issue template? It makes sure no special guioptions are set.
2. Also can you show me what this command in Terminal shows (defaults read org.vim.MacVim | grep MM)?
3. Related to (2), but do you have Core Text Renderer disabled?
4. What system languages do you have set, and their relative order? In particular, I'm asking for the list of languages specified in System Preferences → Language & Region. The languages there actually affect the way the OS substitutes fonts as @eirnym was alluding to above.
5. What happens if you put a cursor on the third and fourth column? What's in between them? Does the cursor become super wide?

I don't think any code in the relevant parts had changed in the release so there shouldn't have been much change, so I'm a little surprised to see any rendering differences.

[ychin]

In screenshot below, rightmost right arrow on a line above emoji takes the same width as w above as these symbols are taken from Fira Code Light I use. On the other hand, some non-English, Japanese, "Powerline" and Emoji characters are taken from fallback system fonts, so could have their own width (which is not exactly the same as English letters). You can see this discrepancy in "Powerline" section: lines are looks different despite they have exactly the same amount of characters in them. Similar situation you'll see in emoji: first emoji is in the same column as arrow I mentioned above, but way wider.

The width of a character is always either 1 or 2 in Vim. It's calculated by Vim (not MacVim), and not determined by the font at all. In fact, if the two (Vim versus font) don't agree, that's usually when you would get weird artifacts.

[eirnym]

The width of a character is always either 1 or 2 in Vim. It's calculated by Vim (not MacVim), and not determined by the font at all. In fact, if the two (Vim versus font) don't agree, that's usually when you would get weird artifacts.

Thank you for describing this, I took it from "cut emoji" issue.

[shikato]

1

I tried mvim --clean, but same issue occurred.

[shikato]

2

[shikato]

3

Core Text Renderer off

The issue occurred.

Core Text Renderer on

The issue did not occur. (I noticed it in this test.)

[shikato]

4

language order

1. Japanese
2. English

and location is Japan.

[shikato]

5

The explanation is very difficult, so I recorded a video. Can you watch it? @ychin https://user-images.githubusercontent.com/4592677/192508749-0b0a823d-74b1-4483-8948-2afbec7b9bbb.mov

[eirnym]

@shikato yes, video explained issue quite better

How did you install MacVim? I have MacVim installed from GitHub releases and have no such issue via copy and paste and direct input (I installed standard japanese keyboard for the test via system preferences)

[shikato]

@eirnym I installed MacVim from github release notes dmg file's link

I found a tweet, maybe it is same issue, too. https://twitter.com/sat_toke/status/1572623078760521730

[ychin]

@shikato , out of curiosity, why are you setting Core Text renderer to off? As the description there says, this is deprecated, meaning that using MacVim without Core Text renderer is no longer supported (and we no longer fix bugs / glitches associated with it). I still exposed the option for now in case we need to debug issues, but it would be removed in the near future with no option to turn it off. Can you just use it with Core Text renderer on? What feature do you find missing?

[ychin]

I'm still curious what broke though, so I'll take a look, but you should probably assume that Core Text Renderer off will break at any time.

[ychin]

I think this was caused by #1287 just for reference.

[shikato]

@ychin I have a hazy memory, I think the reason was to avoid similar issue. https://github.com/macvim-dev/macvim/issues/1168#issuecomment-766273510

I think the issue doesn't occur in current version, so I can enable Core Text renderer.

Thank you for confirming.

[ychin]

Right. Yes please enable Core Text renderer. I will fix this bug anyway since it's easy to fix, but I should probably just remove the option to unset Core Text renderer in a future release to avoid stale settings like this leading to obscure bugs. Feel free to file additional issues if you find other issues.

[sattoke]

@shikato

I found a tweet, maybe it is same issue, too. https://twitter.com/sat_toke/status/1572623078760521730

Thank you for referring to my tweet.

@ychin I enabled "Use Core Text renderer" and it solved the problem in my environment as well! Thank you!

In my environment, with that option enabled, MacVim (I don't remember the version) did not display properly in the former Big Sur environment, such as the display blinking, so I disabled it. https://twitter.com/sat_toke/status/1368800041708650497

[ychin]

I see. Thanks for the context. I'm probably going to remove the option to unset the core Text renderer in the future. If we run into more displays issues we will just have to fix them instead of allowing a poor workaround (I still think it was likely due to the MacVim version being old and didn't have the latest fixes to the renderer).