Slow for large dictionaries

[fenuks]

I use two dictionaries, one for language with inflection which means number of entries is huge (4493307 vs 127466 for British English, which means it is over 35 times larger). cmp-dictionary slows down nvim startup by little more than six seconds for me. I took a look at source, and it seems that file is read asynchronously, yet it blocks the editor. I'm using nvim v0.5.1.

[uga-rosa]

I'll look into it. Can you tell me where I can download the dictionary to reproduce it?

[fenuks]

If you are using Arch-based distro, you can install it from AUR. Otherwise, you could generate it this way (it required dos2unix and unzip utilities):

wget http://web.archive.org/web/20210604075633/https://sjp.pl/slownik/odmiany/sjp-odm-20210427.zip -O dict.zip
unzip dict.zip
dos2unix odm.txt
grep -v '^\?' odm.txt | sed 's/, /\n/g' | sort > polish

Alternatively, I guess you could also concatenate the dictionary you are using some 30 times. You would get a lot of duplicated entries, but that doesn't matter here.

[uga-rosa]

I was able to reproduce it. I will investigate the cause.

[uga-rosa]

I found the cause, but I don't know the solution, so it may take some time. vim.loop's timer doesn't seem to be truly asynchronous. Try this snippet to see if it helps.

local function sleep(n)
    os.execute("sleep " .. n)
end

local timer = vim.loop.new_timer()
timer:start(0, 100, function()
    sleep(3)
    timer:close()
end)

[fenuks]

Try this snippet to see if it helps.

It blocks. If I understand luv correctly, problem in this snippet is that os.execute blocks event loop/nvim. If asynchronous version of os.execute was called, then it would run asynchronously without blocking the editor.

[uga-rosa]

This is just an example (because I don't use os.execute in this plugin), and the problem is that the operation is blocked while the function passed to timer is being executed.

[uga-rosa]

@fenuks Can you try #7?

[fenuks]

Yes, this improves startup time dramatically. I had to install lua51-mpack (it seems that neovim nightly requires it, but stable version does not). I've been reading a bit about async in neovim, and stumbled upon this article (describes Lua module written in rust to achieve true async).

[uga-rosa]

I'm using nightly and mpack worked without doing anything special. I don't want to put more dependencies than the mainframe...

[uga-rosa]

7 is a good solution?

[fenuks]

That PR does fix my issue. It might not work for people running stable without mspack installed, so you might want to wait for new neovim release (if I remember correctly, new version will be released tomorrow), or add note it needs to be installed separately.

[uga-rosa]

Ok. Let's see if it works with 0.6, which will be released tomorrow.

[uga-rosa]

merged. Please check the readme for optional activation.