double-width character issue in Nerdtree and Netrw

[hankook]

• Neovim version: 0.1.1 and 0.1.3-361-g5730ad9
• Operating system: OS X el capitan
• Terminal emulator: iTerm2.app and Terminal.app
• Shell : zsh and bash

  Actual behaviour

When I use Korean(double-width characters) in filename, (I think) NEOVIM cannot compute correct length of the filename. So NEOVIM draw horizontal bar incorrectly in Nerdtree.

However, VIM works in this case.

(first image is the NEOVIM case and second image is the VIM case)

Steps to reproduce

1. make directory and files with double-width characters. I used 안녕, 테스트(means hello, test resp.)
2. In the directory, open NEOVIM and Nerdtree.

[justinmk]

Does it happen in Vim (what version, exactly?)?

[hankook]

@justinmk my VIM version is 7.4.1090. In VIM, this issue does not happen(see the second image)

[Jun-T]

This happens only on Mac OS X and with filenames in Hangul.

The default file system on OS X uses (a variant of) NFD for filenames, so a Hangul syllable in filename is decomposed into Jamo. For example, 안 is decomposed into U+110B,U+1161,U+11AB, or "e1 84 8b e1 85 a1 e1 86 ab" in UTF-8. Neither vim nor nvim can handle decomposed Hangul correctly. For example, try

$ mkdir tmp && cd tmp
$ touch 안녕
$ ls | od -t x1 -t c
$ ls > nfd.txt
$ vim nfd.txt     # or nvim nfd.txt

The reason that NERDTree works OK on vim is that vim transforms decomposed filenames into precomposed form (NFC). See lines 10265-10275 of src/misc1.c in the vim's git master. https://github.com/vim/vim/blob/master/src/misc1.c#L10265-L10275

It may be possible to fix nvim so that it can handle decomposed Hangul correctly, but it would require some work. Converting into NFC, as in vim, is much easier, and in some sense more useful, because Hangul inputted by users (for searching, etc.) is usually in NFC.

I've created a PR for this: #4488

@hankook Could you please try this PR?

[hankook]

@Jun-T Thanks! It works propely except one case: filename of root directory has Korean.

I have directories and files:

test/
  ee_안녕/
    test.txt
    안녕.txt

When open neovim(nvim) and nerdtree(:NERDTree) In the test directory, it works.

When open neovim and nerdtree In the ee_안녕 directory, however, it does not works. In this case, filenames is something wrong and I cannot open files.

This issue is happend in Netrw and VIM, too.

[Jun-T]

@hankook Thank you for testing. But I can't reproduce your problem. Both nvim with my patch and vim work fine with :NERDTree path/to/test/ee_안녕.

[hankook]

@Jun-T This problem happens only when open Neovim on the ee_안녕 directory and then open Nerdtree( :NERDTree . or :NERDTree).

[Jun-T]

@hankook Sorry, what I wrote might be not clear enough. I did test as you describe, i.e., cd into ee_안녕, start nvim, and run :NERDTree; it works fine for me. Does vim work for you in this case?

[hankook]

@Jun-T No, Vim does not work in this case on my mac.

[hankook]

I made another directory hello in ee_안녕 and put some files in hello, and tried it: open neovim in test/ee_안녕/hello directory and open Nerdtree. In this case, this issue also happened.

I think this issue happens when the directory path contains Korean.

[Jun-T]

@hankook Which version of NERDTree are you using? I'm using their git HEAD. Their change log doesn't show anything which seems to be relevant to the current problem, but just in case could you try the latest NERDTree?

[hankook]

@Jun-T I already tried latest NERDTree(commit: 4ebbb53). This issue happens in Netrw, also. In Netrw, filnames do not appear in netrw list.

Neither neovim nor vim work fine.

[Jun-T]

@hankook Sorry, I can't reproduce your problem; both vim and nvim+PR, and both NERDTree and Netrw work OK when started in the directory whose name contains Hangul. Are there anything in your init.vim which may interfere?

[hankook]

@Jun-T Um... My init.vim is very simple. (.vimrc is same as init.vim)

"--------------"
" PLUGINS      "
"--------------"

call plug#begin("~/.nvim/plugged")

Plug 'altercation/vim-colors-solarized'
"Plug 'LaTeX-Box-Team/LaTeX-Box'
"Plug 'bling/vim-airline'
"Plug 'tpope/vim-fugitive'
Plug 'scrooloose/nerdtree'
"Plug 'jistr/vim-nerdtree-tabs'
"Plug 'jlanzarotta/bufexplorer'
"Plug 'Shougo/unite.vim'
"Plug 'ctrlpvim/ctrlp.vim'
"Plug 'Shougo/vimfiler.vim'

call plug#end()

"--------------"
" VIEW         "
"--------------"

set backspace=indent,eol,start

set t_Co=256
colorscheme solarized
set background=dark

let g:airline_left_sep=''
let g:airline_right_sep=''

"let g:nerdtree_tabs_open_on_console_startup=1

"--------------"
" SYNTAX       "
"--------------"

syntax on
set autoindent
set expandtab

autocmd Filetype * set ts=2 sts=2 sw=2
autocmd Filetype python set ts=4 sts=4 sw=4
autocmd Filetype java set ts=4 sts=4 sw=4
autocmd Filetype javascript set ts=2 sts=2 sw=2
autocmd Filetype matlab set ts=4 sts=4 sw=4
autocmd Filetype go set ts=4 sts=4 sw=4

set ignorecase
set incsearch
set nohlsearch

"--------------"
" KEY BINDINGS "
"--------------"

let mapleader=','
let maplocalleader=','
noremap ; :

" moving cursor
noremap j gj
noremap k gk

" moving tabs
nnoremap <leader>1 1gt
nnoremap <leader>2 2gt
nnoremap <leader>3 3gt
nnoremap <leader>4 4gt

I built PR in this way: make and used neovim as build/bin/nvim. Is it right?

[hankook]

In current stable Neovim(0.1.1), the filenames appear in nerdtree.

[Jun-T]

I installed OS X 10.11 (El Captain) on an external drive and tested on it, and have found that NERDTree and Netrw do not work well (as hankook describes) if the current directory contains Hangul, although they work fine on 10.9 or 10.8. (I can't test on 10.10 now).

It seems the behavior of the system's library function getcwd(3) has changed between 10.9 and 10.11; it returns NFC on 10.9 but NFD on 10.11.

With my PR, vim functions like glob() or globpath() return NFC, but on 10.11 vim function getcwd() now returns NFD and seems to be causing the problem in NERDTree/Netrw (although I didn't understand the detail). So it seems that converting from NFD to NFC only in some part of the nvim source code is not a good idea. If we want to use NFC, we may need to convert to NFC everywhere in the source.

But, in addition to doing so is quite tedious (if not impossible), there is another reason that we can not use NFC always. For example, there is a function path_fix_case() in path.c, which is important on OS X since the default file system is case insensitive (but case preserving). Currently this function doesn't work well on OS X for file names with accent. On OS X, über.txt and Über.txt are the same file, but this is not detected by path_fix_case(), because one of the name comes from user (command line) and in NFC, while the other is from the file system and in NFD. Converting both names into NFC does not work, because ü is not the lowercase of Ü in NFC; we must convert both into NFD for STRICMP to work properly.

As a result, if there is a file über.txt, and edit it by $ nvim Über.txt # or vim Über.txt and save it from within (n)vim, then the file name changes to Über.txt.

I guess fixing all the NFC/NFD related problems is quite a lot of work. NFD is more appropriate for comparison etc., while NFC is easier to display (counting the number of cells etc.). Possibilities may be:

(1) Use NFC file names as default, and convert to NFD only where it is required (for comparison etc.) (2) Use NFD everywhere, and fix nvim so that it can correctly display Hangul in NFD (it can already correctly display other decomposed characters such as accented alphabets). (3) ....(other possibilities?)

but anyway it would be a lot of work...

@hankook I think the current nvim (without my PR) is "usable" with NERDTree/Netrw although there are some display problem (did you try ctrl-L which may improve the display?). If you agree with this, applying my PR is not a good idea, because NERDTree/Netrw becomes "unusable" if the current directory has Hangul in it (on OS X 10.11).

[hankook]

@Jun-T I understand. I have same opinion: current nvim is more usable.

If cursor in NERDTree is not on a line containing no Korean, Ctrl-L or changing tab work properly. Otherwise, these not work.

I think placing NERDTree on the right(let NERDTreeWinPos='right') is the best solution for this issue.

Thank you :-)

[qvacua]

As of v0.3.4 this problem is not limited to file names on macOS. Neovim seems to handle decomposed UTF-8, i.e. NFD, strings incorrectly. For example the hex representation of 한글 in NFD is

e18492e185a1e186abe18480e185b3e186af

If you open a file with above content, then the following chunks are passed from ui_line(...) to ui_raw_line(...)

[[e1,84,92], [], [e1,85,a1], [e1,86,ab], [e1,84,80], [], [e1,85,b3], [e1,86,af]]

where [] denotes the empty cell for wide characters. This has wrong cell number and also wrong [] placement. It should be

[[e1,84,92, e1,85,a1, e1,86,ab], [], [e1,84,80, e1,85,b3, e1,86,af], []]

or if we want to only use NFC

[[ed,95,9c], [], [ea,b8,80], []]

iTerm2 seem to render it correctly somehow, but if you move the cursor around, you notice that there are more than two (wide) cells.

There was also a brief discussion on Gitter.