Pipe lines directly to `fzy`

[idanarye]

Old behavior was loading all the commands into a Vim list, write it into a file, and pipe that file into fzy.

This is a inefficient, because both ag/rg and fzy are blazing fast and can handle large quantities of data, but Vimscript, with it's dynamic typing and garbage-collected memory - which aren't even state-of-the-art! - is not only slowing the entire process down but also breaks it when doing a big :FuzzyGrep in a large codebases.

This PR aims to fix it:

• With :FuzzyGrep it pipes the ag/rg command directly to fzy.
• With :FuzzyOpen it creates a temporary file, writes the list of buffers to it with writefile() and then uses ag/rg to pipe the list of files into that temporary file without passing them through Vim.

This ensures that aside from the list of buffers(that for obvious reasons must come from Vim) all the match choices are piped by the operation system and Vim does not need to load them into it's memory.

[idanarye]

I tested it by running :FuzzyGrep without any arguments from my /usr/include. Without this commit it took about 6 seconds until fzy shows up with a list - with this commit it took 1 second. Currently downloading the Linux kernel to test it on a really big project.

[idanarye]

OK, with the Linux kernel it takes 12 seconds to :FuzzyGrep with current master and 2 seconds with this commit.

[cloudhead]

Very nice, thanks, I will look into this!

[insidewhy]

Tested out this change on a pretty large project I'm working on and it is great.

[cloudhead]

Hmm is there a way not to lose the choice count in the status line?

[idanarye]

Calling ripgrep twice will still be faster than loading the results into Vim. Either that or we can put them in a file.

I'm pondering about how to count the lines though, as we can't load them into Vim. Linux has wc -l and Windows has find /c, but I hate splitting the implementation more than I have to - so maybe it would be better to use rg -c for both? Then again, wc -l seems to be about 15% faster than rg -c, so it may be worth it...

[insidewhy]

I'm wondering if you can hook into the terminal API to get the count. A double parse would be preferable to viml's slowness but it should be possible to have the best of both worlds. Using jobs and interpreting the results in python was how I was doing it. Involved tonnes of boilerplate to manage windows and bindings though, despite the added flexibility and speed I switched to fuzzy due to the terseness of the code.

[idanarye]

What good will hooking to the terminal API do? It's not like fzy shows the number of matches in it's TUI...

[insidewhy]

If you could have fzy put all the results into the terminal and do the clipping inside of the plugin then the count could be read. Would need to also hook into the up/down commands for scrolling the end of the buffer though so introduces some complexity.

[idanarye]

You can set the number of lines fzy prints to be very high - but you still need to specify a limit. Neovim's terminal also has a limit of lines(before it deletes old ones) - not sure if you can even change it.

Also, fzy number of lines is limited by the terminal's height, and even if we can somehow fake this I'm not sure how it will affect Neovim's terminal...

But even if we could solve all these issues - the overhead of printing lines to a terminal is pretty high, and it may even be worse than loading all the lines to Vimscript...

[cloudhead]

Hmm it's definitely not worth it if we have to run an extra command I would say. The speed benefit of piping outweighs that. Something like tee or pv -l could be useful here though pv isn't installed on most systems.

[idanarye]

You mean something like this?

rg '<my-regex>' | tee /tmp/tmpfile | wc -l

I'll try to experiment with this, if writing the file has any impact on performance.

Another option is:

rg '<my-regex>' > /tmp/tmpfile
wc -l /tmp/tmpfile

It won't be that different from using tee, and it'll be more directly translatable to Windows.

[idanarye]

In Linux we can use FIFOs:

mkfifo /tmp/myfifo

The we'll start a terminal with

rg '<my-regex>' | tee /tmp/myfifo | fzy

And immediately count the lines with:

wc -l /tmp/myfifo

Windows does not have FIFOs. It has named pipes, but AFAIK you can't use them from the shell(unless it's PowerShell, which is a fully blown .NET scripting language with shell-like syntax...). We'll have to use a file, or call ripgrep twice. Oh well - Windows users should be used to inferior development tools...

[insidewhy]

Windows users should be used to inferior development tools..

I've been using Linux for 20 years now but this statement... Visual Studio is still amazing.

[cloudhead]

Windows support is not something I'm very concerned with -- do rg/ag even work on windows?

[idanarye]

OK, I tried doing it on windows. rg works on Windows. fzy though... not so much. I tried with fzf, but encountered another problem - Neovim's terminal support is extremely poor. At least on my setup. Might have something to do with lack of proper PTY...

So, yea - Windows support is not something this plugin currently can have, with or without this PR.

Anyways, I've implemented the FIFO solution(Linux only - will probably work on Mac but I can't check it) and it works fine.

[cloudhead]

This seems to break file search/file open when not in the root folder of a git project. I haven't figured out why yet. To reproduce, cd into a sub-folder of a git project, run nvim with no arguments, run :FuzzyOpen then notice it only shows files in that folder. Then try to open one, it will be empty. Any clues?

[cloudhead]

Update: the problem is due I think to the lcd commands not affecting rg anymore, I'm guessing because it's run differently. Not sure how to fix yet.