neovim 加载大词库速度太慢了

jdhao commented 3 years ago

试了一下你提供的 pinyin_huge.txt 作为 db 文件，我掐表计时，从输入 nvim 命令，到显示 buffer 内容，需要大约 30 秒。

ZSaberLv0 commented 3 years ago

https://github.com/ZSaberLv0/ZFVimIM#known-issue 请先确保:

executable('python') 或者 executable('python3')
ZFJobAvailable()

正常配置下, pinyin_huge 加载时间应该在 10 秒左右, 并且不会阻塞界面

jdhao commented 3 years ago

python3 是有的， ZFJob 那个插件也安装了。系统用的是 centos 7。等会我看一下 mac 上能不能复现

ZSaberLv0 commented 3 years ago

加了个自检函数, 请更新一下然后 call ZFVimIM_DEBUG_checkHealth()

jdhao commented 3 years ago

在 macOS 上测了一下，也是同样的问题，加载是 blocking 的，加载 pinyin_huge.txt 需要 25 s 左右。

call ZFVimIM_DEBUG_checkHealth() 输出：

ZFJobAvailable: 1                                                                                                                                                                                       
    vim version: 800
    vim job: 0
    nvim job: 1
python: 1
    python: 1 /Users/jdhao/anaconda3/bin/python
        Python 3.7.0
    python3: 1 /Users/jdhao/anaconda3/bin/python3
        Python 3.7.0

ZSaberLv0 commented 3 years ago

提供以下信息:

min.vim

filetype plugin indent on
syntax on
set nocompatible
let g:plug_home = $HOME . '/.vim/bundle'
let g:plug_url_format = 'https://github.com/%s'
execute 'source ' . g:plug_home . '/vim-plug/plug.vim'
silent! call plug#begin()
Plug 'junegunn/vim-plug'

Plug 'ZSaberLv0/ZFVimIM'
Plug 'ZSaberLv0/ZFVimIM_pinyin_huge'
Plug 'ZSaberLv0/ZFVimJob'

set laststatus=2
let g:ZFVimIM_DEBUG_profile = 1
let g:ZFJobVerboseLogEnable = 1
call plug#end()

nvim -u min.vim, 并等词库加载完成
提供 call ZFVimIM_DEBUG_profileInfo()
call writefile(g:ZFJobVerboseLog, 'log.txt'), 提供 log 文件

jdhao commented 3 years ago

使用上述 min.vim

提供 call ZFVimIM_DEBUG_profileInfo()

dbLoadPart avg:2 (68/23) max:6 min:0

call writefile(g:ZFJobVerboseLog, 'log.txt'), 提供 log 文件

log 文件内容如下：

23:51:22 groupJob 1 start
23:51:22 groupJob 1 running group 0
23:51:22 job 1 start: `python3 "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbLoad.py" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge/misc/pinyin_huge.txt" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge/misc/pinyin_huge_count.txt" "/Users/jdhao/.vim_cache/ZFVimIM/ZFVimIM_dbLoad_1"`
23:51:22 groupJob 1 23:51:22 job 1 start: `python3 "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbLoad.py" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge/misc/pinyin_huge.txt" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge/misc/pinyin_huge_count.txt" "/Users/jdhao/.vim_cache/ZFVimIM/ZFVimIM_dbLoad_1"`
23:51:23 job 1 output [stderr]: 
23:51:23 groupJob 1 23:51:23 job 1 output [stderr]: 
23:51:23 job 1 output [stdout]: 
23:51:23 groupJob 1 23:51:23 job 1 output [stdout]: 
23:51:23 job 1 stop with exitCode 0: `python3 "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbLoad.py" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge/misc/pinyin_huge.txt" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge/misc/pinyin_huge_count.txt" "/Users/jdhao/.vim_cache/ZFVimIM/ZFVimIM_dbLoad_1"`
23:51:23 groupJob 1 23:51:23 job 1 stop with exitCode 0: `python3 "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbLoad.py" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge/misc/pinyin_huge.txt" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge/misc/pinyin_huge_count.txt" "/Users/jdhao/.vim_cache/ZFVimIM/ZFVimIM_dbLoad_1"`
23:51:23 groupJob 1 running group 1
23:51:23 job 2 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 2 start: `sleep 10ms`
23:51:23 job 3 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 3 start: `sleep 10ms`
23:51:23 job 4 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 4 start: `sleep 10ms`
23:51:23 job 5 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 5 start: `sleep 10ms`
23:51:23 job 6 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 6 start: `sleep 10ms`
23:51:23 job 7 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 7 start: `sleep 10ms`
23:51:23 job 8 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 8 start: `sleep 10ms`
23:51:23 job 9 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 9 start: `sleep 10ms`
23:51:23 job 10 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 10 start: `sleep 10ms`
23:51:23 job 11 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 11 start: `sleep 10ms`
23:51:23 job 12 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 12 start: `sleep 10ms`
23:51:23 job 13 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 13 start: `sleep 10ms`
23:51:23 job 14 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 14 start: `sleep 10ms`
23:51:23 job 15 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 15 start: `sleep 10ms`
23:51:23 job 16 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 16 start: `sleep 10ms`
23:51:23 job 17 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 17 start: `sleep 10ms`
23:51:23 job 18 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 18 start: `sleep 10ms`
23:51:23 job 19 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 19 start: `sleep 10ms`
23:51:23 job 20 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 20 start: `sleep 10ms`
23:51:23 job 21 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 21 start: `sleep 10ms`
23:51:23 job 22 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 22 start: `sleep 10ms`
23:51:23 job 23 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 23 start: `sleep 10ms`
23:51:23 job 24 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 24 start: `sleep 10ms`
23:51:23 job 25 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 25 start: `sleep 10ms`
23:51:23 job 26 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 26 start: `sleep 10ms`
23:51:23 job 27 start: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 27 start: `sleep 10ms`
23:51:23 job 2 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 2 stop with exitCode 0: `sleep 10ms`
23:51:23 job 3 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 3 stop with exitCode 0: `sleep 10ms`
23:51:23 job 4 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 4 stop with exitCode 0: `sleep 10ms`
23:51:23 job 5 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 5 stop with exitCode 0: `sleep 10ms`
23:51:23 job 6 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 6 stop with exitCode 0: `sleep 10ms`
23:51:23 job 7 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 7 stop with exitCode 0: `sleep 10ms`
23:51:23 job 8 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 8 stop with exitCode 0: `sleep 10ms`
23:51:23 job 9 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 9 stop with exitCode 0: `sleep 10ms`
23:51:23 job 10 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 10 stop with exitCode 0: `sleep 10ms`
23:51:23 job 11 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 11 stop with exitCode 0: `sleep 10ms`
23:51:23 job 12 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 12 stop with exitCode 0: `sleep 10ms`
23:51:23 job 13 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 13 stop with exitCode 0: `sleep 10ms`
23:51:23 job 14 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 14 stop with exitCode 0: `sleep 10ms`
23:51:23 job 15 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 15 stop with exitCode 0: `sleep 10ms`
23:51:23 job 16 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 16 stop with exitCode 0: `sleep 10ms`
23:51:23 job 17 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 17 stop with exitCode 0: `sleep 10ms`
23:51:23 job 18 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 18 stop with exitCode 0: `sleep 10ms`
23:51:23 job 19 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 19 stop with exitCode 0: `sleep 10ms`
23:51:23 job 20 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 20 stop with exitCode 0: `sleep 10ms`
23:51:23 job 21 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 21 stop with exitCode 0: `sleep 10ms`
23:51:23 job 22 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 22 stop with exitCode 0: `sleep 10ms`
23:51:23 job 23 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 23 stop with exitCode 0: `sleep 10ms`
23:51:23 job 24 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 24 stop with exitCode 0: `sleep 10ms`
23:51:23 job 25 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 25 stop with exitCode 0: `sleep 10ms`
23:51:23 job 26 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 26 stop with exitCode 0: `sleep 10ms`
23:51:23 job 27 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 23:51:23 job 27 stop with exitCode 0: `sleep 10ms`
23:51:23 groupJob 1 stop [0]
23:51:23 groupJob 2 start
23:51:23 groupJob 2 running group 0
23:51:23 job 28 start: `sh "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbDownload.sh" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge" "" "" ""`
23:51:23 groupJob 2 23:51:23 job 28 start: `sh "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbDownload.sh" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge" "" "" ""`
23:51:23 job 28 output [stderr]: Updated 0 paths from the index
23:51:23 groupJob 2 23:51:23 job 28 output [stderr]: Updated 0 paths from the index
23:51:23 job 28 output [stdout]: Fetching origin
23:51:23 groupJob 2 23:51:23 job 28 output [stdout]: Fetching origin
23:51:25 job 28 output [stdout]: HEAD is now at e04470a update
23:51:25 groupJob 2 23:51:25 job 28 output [stdout]: HEAD is now at e04470a update
23:51:26 job 28 output [stdout]: Already up to date.
23:51:26 groupJob 2 23:51:26 job 28 output [stdout]: Already up to date.
23:51:26 job 28 output [stderr]: 
23:51:26 groupJob 2 23:51:26 job 28 output [stderr]: 
23:51:26 job 28 output [stdout]: 
23:51:26 groupJob 2 23:51:26 job 28 output [stdout]: 
23:51:26 job 28 stop with exitCode 0: `sh "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbDownload.sh" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge" "" "" ""`
23:51:26 groupJob 2 23:51:26 job 28 stop with exitCode 0: `sh "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbDownload.sh" "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM_pinyin_huge" "" "" ""`
23:51:26 groupJob 2 stop [0]

jdhao commented 3 years ago

发现了一些端倪。

使用上面你给的 min.vim，nvim 不会卡住，加载很快，并且 ZFVimIM 可以正常使用。
我用的加载速度慢的配置，和插件主页上给的稍微有点不同，用到的 autocmd 事件不同，完整配置如下：
```
call plug#begin(expand('~/.local/share/nvim/plugged'))
Plug 'ZSaberLv0/ZFVimIM'
Plug 'ZSaberLv0/ZFVimJob'
call plug#end()
```

function! s:myLocalDb() let db = ZFVimIM_dbInit({ \ 'name' : 'jd_db', \ }) call ZFVimIM_cloudRegister({ \ 'mode' : 'local', \ 'dbId' : db['dbId'], \ 'repoPath' : expand('./'), \ 'dbFile' : 'pinyin_huge.txt', \ }) endfunction

autocmd VimEnter * call s:myLocalDb()



使用这个配置打开 nvim，速度很慢，nvim 加载好之后，可以正常使用 ZFVimIM 输入法

3. 把上面的配置中  `autocmd VimEnter * call s:myLocalDb()` 改成插件主页提供的方式 `autocmd User ZFVimIM_event_OnDbInit call s:myLocalDb()`, 其他不变，nvim 加载很快，但是在 status 上会看到  db update fail，exitCode 1 的信息提示。然后 ZFVimIM 输入法也使用不了了（按 `;;` 然后输入字符，不会出现补全窗口）

jdhao commented 3 years ago

经过各种 debug，发现了原因，dbCountFile 这一项并不是 optional，如果没有写这一项，经过 debug，发现 log.txt 内容如下：

01:22:50 groupJob 1 start
01:22:50 groupJob 1 running group 0
01:22:50 job 1 start: `python3 "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbLoad.py" "/Users/jdhao/Projects/trial_error/pinyin_huge.txt" "/Users/jdhao/Projects/trial_error" "/Users/jdhao/.vim_cache/ZFVimIM/ZFVimIM_dbLoad_1"`
01:22:50 groupJob 1 01:22:50 job 1 start: `python3 "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbLoad.py" "/Users/jdhao/Projects/trial_error/pinyin_huge.txt" "/Users/jdhao/Projects/trial_error" "/Users/jdhao/.vim_cache/ZFVimIM/ZFVimIM_dbLoad_1"`
01:22:51 job 1 output [stderr]: Traceback (most recent call last):
01:22:51 groupJob 1 01:22:51 job 1 output [stderr]: Traceback (most recent call last):
01:22:51 job 1 output [stderr]:   File "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbLoad.py", line 13, in <module>
01:22:51 groupJob 1 01:22:51 job 1 output [stderr]:   File "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbLoad.py", line 13, in <module>
01:22:51 job 1 output [stderr]:     pyMap = dbFunc.dbLoadPy(DB_FILE, DB_COUNT_FILE)
01:22:51 groupJob 1 01:22:51 job 1 output [stderr]:     pyMap = dbFunc.dbLoadPy(DB_FILE, DB_COUNT_FILE)
01:22:51 job 1 output [stderr]:   File "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbFunc.py", line 130, in dbLoadPy
01:22:51 groupJob 1 01:22:51 job 1 output [stderr]:   File "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbFunc.py", line 130, in dbLoadPy
01:22:51 job 1 output [stderr]:     with io.open(dbCountFile, 'r', encoding='utf-8') as dbCountFilePtr:
01:22:51 groupJob 1 01:22:51 job 1 output [stderr]:     with io.open(dbCountFile, 'r', encoding='utf-8') as dbCountFilePtr:
01:22:51 job 1 output [stderr]: IsADirectoryError: [Errno 21] Is a directory: '/Users/jdhao/Projects/trial_error'
01:22:51 groupJob 1 01:22:51 job 1 output [stderr]: IsADirectoryError: [Errno 21] Is a directory: '/Users/jdhao/Projects/trial_error'
01:22:51 job 1 output [stderr]: 
01:22:51 groupJob 1 01:22:51 job 1 output [stderr]: 
01:22:51 job 1 output [stdout]: 
01:22:51 groupJob 1 01:22:51 job 1 output [stdout]: 
01:22:51 job 1 stop with exitCode 1: `python3 "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbLoad.py" "/Users/jdhao/Projects/trial_error/pinyin_huge.txt" "/Users/jdhao/Projects/trial_error" "/Users/jdhao/.vim_cache/ZFVimIM/ZFVimIM_dbLoad_1"`
01:22:51 groupJob 1 01:22:51 job 1 stop with exitCode 1: `python3 "/Users/jdhao/.local/share/nvim/plugged/ZFVimIM/misc/dbLoad.py" "/Users/jdhao/Projects/trial_error/pinyin_huge.txt" "/Users/jdhao/Projects/trial_error" "/Users/jdhao/.vim_cache/ZFVimIM/ZFVimIM_dbLoad_1"`
01:22:51 groupJob 1 stop [1]

因为缺少了 dbCountFile , 导致 DbFunc.py 计算出来的 dbCountFile 的路径不对，所以出错了。解决办法就是在 db 初始化的时候加上 dbCountFile 这个 key 以及对应的文件。

下面的配置可以正常运行

call plug#begin(expand('~/.local/share/nvim/plugged'))
  Plug 'ZSaberLv0/ZFVimIM'
  Plug 'ZSaberLv0/ZFVimJob'
call plug#end()

function! s:myLocalDb()
  let db = ZFVimIM_dbInit({
              \   'name' : 'jd_db',
              \ })
    call ZFVimIM_cloudRegister({
                \   'mode': 'local',
                \   'dbId' : db['dbId'],
                \   'repoPath' : expand('./'),
                \   'dbFile' : 'pinyin_huge.txt',
                \   'dbCountFile': 'pinyin_huge_count.txt'
                \ })
endfunction

let g:ZFVimIM_DEBUG_profile = 1
let g:ZFJobVerboseLogEnable = 1

" autocmd VimEnter * call s:myLocalDb()
autocmd User ZFVimIM_event_OnDbInit call s:myLocalDb()

ZSaberLv0 commented 3 years ago

大E了, 没有闪已修复

ZSaberLv0 / ZFVimIM

neovim 加载大词库速度太慢了 #23