fix: dir/filename is broken (at least) in Korean

jjangga0214 commented 1 year ago

Discord username (optional)

No response

Describe the bug

When file/directory names are in Korean, they become broken in warp.

I executed ls, and the result is like the above screenshot. The original name of the directory was "ai-검증", while printed as "ai-ㄱㅈ". It seems like only the initial letters of each block(Korean hangeul has a concept of 'block') are printed. This behavior does not happen with other terminals(e.g. iterms 2, tabby, etc).

To Reproduce

Create a file/directory name in Korean and execute ls.

Expected behavior

No response

Screenshots

No response

Operating System

MacOS

Operating System and Version

Ventura 13.3.1 (a)(22E772610a)

Shell Version

No response

Warp Version

No response

Additional context

No response

Does this block you from using Warp daily?

Yes, this issue prevents me from using Warp daily.

Is this a Warp specific issue? (i.e. does it happen in Terminal, iTerm, Kitty, etc.)

Yes, this I confirmed this only happens in Warp, not other terminals.

Warp Internal (ignore): linear-label:b8107fdf-ba31-488d-b103-d271c89cac3e

None

dannyneira commented 1 year ago

Hi @jjangga0214 Sorry to hear about this blocker. I'm not seeing the same on my end when using the default Font, are you using any special font by any chance?

"ai-검증" shows up as normal when I run ls

CleanShot 2023-05-08 at 16 20 35

jjangga0214 commented 1 year ago

Thank you for the reply. This is my setting, which I have not touched.

It's strange. I have no idea.

dannyneira commented 1 year ago

I think this may be related to the lack of International keyboard support. So maybe when you enter the characters it's a different code than with US keyboards. See this issue for more details > #341

vorporeal commented 1 year ago

@jjangga0214 Can you provide a mkdir command we can run to reproduce the issue?

jjangga0214 commented 1 year ago

@vorporeal

https://github.com/warpdotdev/Warp/assets/28584151/c40e12f3-6b05-4b56-afb7-b2e301649795

I found an interesting phenomenon. It works fine when I create(mkdir) a directory through Warp. But the issue happens against directories made in macOS Finder.

gmlwhdtjd commented 1 year ago

@vorporeal

I'm experiencing the same issue, so here are some additional oddities

Even if the file was created with mkdir, I could see that the filename was broken just by entering the name modification in finder.

https://github.com/warpdotdev/Warp/assets/25698134/aa61b332-17b1-4d60-a963-067842a0803d

hansolsong commented 1 year ago

I think this is a bug caused by a mismatch in Unicode Normalization Forms between macOS and Warp.

OS X normalizes unicode filenames using NFD(Normalization Form Canonical Decomposition), while every other major platform uses NFC(Normalization Form Canonical Composition).

It seems that Warp treats unicode strings to be normalized using NFC, so when you create a file/directory inside Warp then its name is displayed properly (albeit not conforming to OS behavior), but it breaks when you deal with files that are created or renamed outside Warp (thus having names normalized using NFD).

Below is a demonstration of this behavior, converting the filename between two forms. (Please ignore incorrect file counts.)

warpdotdev / Warp