Slow initial scan

[mgedmin]

For some reason command-t has been very slow to do the initial scan with recent versions of vim. I press \t and I see it scanning files, then it stops for a long while at 8453 files or so, nothing happens for like 20-40 seconds, and then it unfreezes and I can use my vim again.

Forcing command-t to rescan with :CommandTFlush is fast. Quitting and re-running vim and using \t is fast. Asking the Linux kernel to drop filesystem caches with echo 3 | sudo tee /proc/sys/vm/drop_caches is needed to make the problem reappear.

I'm not sure exactly how many files there are in my tree (I wasn't able to find any diagnostic commands in :help command-t), but some bisetion with g:CommandTMaxFiles shows that there are more than 10000 but less than 10500.

I'm on Ubuntu 19.10 and vim 8.1.2291.

[mgedmin]

I did an strace and I see the vim process is busy reading pathnames inside my node_modules from a pipe. And I do have let g:CommandTFileScanner = "find" in my .vimrc.

Basically, there's nothing command-t can do about it. (Well, it could ask find not to descend into node_modules but parsing the wildcards in &wildignore to construct -name node_modules -prune -o bits on the right place of the find command line seems complicated.)

[wincent]

One thing you might consider trying @mgedmin is using the Watchman scanner. It would require you to install Watchman, but it's a tool that is designed to index file hierarchies way in excess of 10k files without skipping a beat. I'd expect it would handle the kind of workload that you describe quite easily.

[mgedmin]

Actually, it is command-t's fault after all!

$ echo 3 |sudo tee /proc/sys/vm/drop_caches > /dev/null
$ time find -type f|wc -l
242228

real    0m6,485s
user    0m0,603s
sys 0m2,858s

6 seconds to find the files.

$ time vim +CommandT
# press Esc Esc Esc ZQ while command-t is scanning
real    0m49,698s
user    0m6,366s
sys 0m19,220s

50 seconds (! not 10-20!) to process them.

Also I haven't cleaned the FS caches after the find and before vim +CommandT.

A second run is faster:

$ time vim +CommandT
# press Esc Esc Esc ZQ while command-t is scanning
real    0m1,588s
user    0m1,303s
sys 0m0,901s

I wonder why! I thought find would already cache everything that's necessary...

[wincent]

Yeah, not sure why there'd by a difference like that. On macOS in a big repo I see this when using the "find" scanner:

$ time find . -type f|wc -l
  441719
find . -type f  1.39s user 8.29s system 39% cpu 24.627 total
wc -l  0.04s user 0.02s system 0% cpu 24.626 total
$ time vim +CommandT
nvim +CommandT  4.64s user 8.99s system 52% cpu 25.950 total

So, roughly the same inside and out. I don't expect it makes a difference, but the actual find command run by Command-T ends up being something like this (from here):

find -L . \
  -mindepth 1 \
  -maxdepth 15 \
  -type f \ 
  -print0 \
  -o -name .* -prune