Always No results with ripgrep 13.0

[trsdln]

After upgrade to Ripgrep 13.0 constantly shows "No results". Visually output or behavior of ripgrep hasn't changed.

[blueyed]

Confirmed.

It hangs with/at read(0, when looking at / wrapping it with strace -f.

Using https://github.com/mhinz/vim-grepper/commit/9b62e6bdd9de9fe027363bbde68e9e32d937cfa0 also for Linux fixes it, i.e. use e.g. (which also adds --sort-files):

let g:grepper = extend(get(g:, 'grepper', {}), {
…
      \ 'rg': {'grepprg': 'rg -H --no-heading --vimgrep --sort-files $* .'},
      \ })

[blueyed]

See https://github.com/BurntSushi/ripgrep/issues/1892.

[blueyed]

Note that using $* . is not really a fix, given that you then cannot specify directories to search for only, i.e. foo bar would search for "foo" in "bar" and "." (see https://github.com/mhinz/vim-grepper/issues/162).

In https://github.com/BurntSushi/ripgrep/issues/1892#issuecomment-860526360 @kevinhwang91 mentioned to use pty=1 with jobstart, which might be more feasible. (Note that adding call jobclose(s:id, 'stdin') to close stdin does not help, because it happens too late apparently)

[datanoise]

It seems that this change fixes the problem:

diff --git a/plugin/grepper.vim b/plugin/grepper.vim
index 73e4e36..7d31877 100644
--- a/plugin/grepper.vim
+++ b/plugin/grepper.vim
@@ -935,6 +935,7 @@ function! s:run(flags)
           \ 'on_stdout': function('s:on_stdout_nvim'),
           \ 'on_stderr': function('s:on_stdout_nvim'),
           \ 'on_exit':   function('s:on_exit'),
+          \ 'pty': 1,
           \ }
     if !a:flags.stop
       let opts.stdout_buffered = 1

[blueyed]

Yeah. Let's wait for feedback on my question at https://github.com/mhinz/vim-grepper/commit/9b62e6bdd9de9fe027363bbde68e9e32d937cfa0. While pty=1 probably has no effect on Windows, maybe the fix for Powershell in ripgrep 0.13 is actually what was meant to be fixed there, so that it could still be reverted then also.

[datanoise]

Actually, using pty=1 requires trimming <CR> characters from the output. And also I'm not sure if it works on Windows at all.

diff --git a/plugin/grepper.vim b/plugin/grepper.vim
index 73e4e36..114a54e 100644
--- a/plugin/grepper.vim
+++ b/plugin/grepper.vim
@@ -155,6 +155,7 @@ function! s:on_stdout_nvim(_job_id, data, _event) dict abort
       " Second-last item is the last complete line in a:data.
       let acc_line = self.stdoutbuf . a:data[0]
       let lcandidates = (empty(acc_line) ? [] : [acc_line]) + a:data[1:-2]
+      let lcandidates = map(lcandidates, {key, val -> trim(val)})
       let self.stdoutbuf = ''
     endif
     " Last item in a:data is an incomplete line (or empty), append to buffer
@@ -935,6 +936,7 @@ function! s:run(flags)
           \ 'on_stdout': function('s:on_stdout_nvim'),
           \ 'on_stderr': function('s:on_stdout_nvim'),
           \ 'on_exit':   function('s:on_exit'),
+          \ 'pty': 1,
           \ }
     if !a:flags.stop
       let opts.stdout_buffered = 1

[insidewhy]

@blueyed Maybe can commit your fix and revert it if/when @mhinz responds? A month is a long time to go without being able to grep in vim :'(

[blueyed]

@insidewhy I think the best fix for now would be https://github.com/mhinz/vim-grepper/issues/244#issuecomment-860755993 from @datanoise. Have you tried it?

As for me, I am holding back the upgrade of ripgrep due to this for now.

I've added some comment/question about an explicit switch to the ripgrep issue: https://github.com/BurntSushi/ripgrep/issues/1892#issuecomment-878253250

[bluz71]

Am I correct to assume that Neovim PR-14812 will fix this issue? No change needed with this plugin?

[jdsutherland]

@bluz71 I just tried neovim HEAD-8baf7bc (after Neovim PR-14812) with vim-grepper HEAD-b80004c and the issue persists. Anyone else?

[bluz71]

@jdsutherland,

I got tired of waiting for a fix, in the end I decided to stop using vim-grepper and just go native (no plugin at all). Native works just as well for me as vim-grepper ever did.

My configuration:

set grepprg=rg\ --vimgrep\ --smart-case
set grepformat=%f:%l:%c:%m,%f:%l:%m

nnoremap <Leader>/ :silent grep<Space>
nnoremap gs :silent grep <C-r><C-w><CR>:copen<CR>
xnoremap gs "sy:silent grep <C-r>s<CR>:copen<CR>

<Leader>/ does a user-prompted ripgrep search and gs does an unprompted search for the word under cursor (or visual selection) whilst also opening the quickfix list. Those mappings are the same as my previous vim-grepper mappings.

For greater control I use fzf.vim :Rg to do interactive ripgrep filtering and selection.

Somehow it feels better to not need to use a plugin at all, my startup is also 2ms faster (haha).

Cheers.

[jdsutherland]

@bluz71 I have a similar setup for fzf.vim but I still use vim-grepper when I want results in quickfix though. There are a few other features unique to vim-grepper keeping me like GrepperOperator, highlight, and async.

[bluz71]

My mappings do populate the quickfix list.

I myself never used GrepperOperator, highlight nor async. Other folks may need those, I didn't.

I could no longer stand waiting for a fix, either on the Neovim side nor here. It could well be that in two months nothing has changed.

[sheldond]

@bluz71 Thanks for posting this! I'm glad to finally have grep via rg working again. I have something to ask you about: for some reason this approach seems to be very slow, there appears to be slowdown while populating the quickfix. Do you see this too?

I apologize if some users feel this is too off-topic, I'd be happy to take it elsewhere.

When comparing the speed of the search mappings to the exact same search ran via :!rg ... I can see the difference in speed. When I run :!rg ... it's very fast, when I run via the mappings it's slow. The slowdown is proportional to the number of results. I was able to replicate with an empty vimrc, so I was wondering do you see the same thing?

p.s. I hacked together a user command that loads the results into quickfix and then also highlights the search string:

command! -nargs=+ QGrep execute 'silent grep! "<args>"' | copen | call feedkeys("/<args><CR>")

nnoremap gs/ :QGrep<Space>

[bluz71]

@sheldond,

I just checked out the rails repository, which is about 300,000 lines of Ruby. Decently big, not the biggest, but bigger than your usual respository.

I did a ripgrep search (via my Neovim mapping) for controller. It took about 1, maybe 2, seconds to find 9,503 matches. I seemed just about instant to me.

Machine, a bog-standard Linux desktop using an SSD.

[sheldond]

@bluz71 Thanks for your response, that is helpful to know! I'm seeing something very similar on rails/rails, the codebase I was working on is about 10x larger than rails so the problem might be more apparent.

When I run rg directly like this :!rg --vimgrep --hidden --follow --max-columns=100 --case-sensitive class it runs very quickly, it's almost instant (< 1 second) regardless of the codebase.

But when I set the same rg command to be the grepprg and then run :grep class it takes about 2 seconds to complete in rails/rails and on the larger codebase it takes a full 28 seconds. This makes me think there is something about vim's grep that is slowing it down. I'm not sure why.

I think this is off-topic so I may not post here about it anymore, but if anyone is interested please feel free to let me know!

[bluz71]

How does Grepper perform by comparison on very large repo?

[bluz71]

@sheldond,

I can reproduce your issue with the Linux Kernal repo. About 16 seconds to search for irq inside Neovim via :grep and about 1.5s in the terminal.

Interesting I also used fzf.vim's :Rg command which highlighted that the ripgrep search was fast, but doing Alt-a to save all matches to the quickfix was very slow.

Hence I think populating the quickfix list with many many items is the bottleneck. That's an upstream issue.

Luckily I never deal with superhuge repos.

[sheldond]

@bluz71 Thanks for sharing the update! It's good to know it's not specific to my system. I was not able to accurately compare with vim-grepper due to this issue with ripgrep v13 (on macos I have not been able to find a way to downgrade ripgrep). It must be quickfix related then. I was able to work-around the issue by finding a way to | head -1000 my results, to limit the number of lines saved into quickfix.

[ghost]

I was not able to accurately compare with vim-grepper due to this issue with ripgrep v13

You can try compare them using other search tool like pt or ag.

on macos I have not been able to find a way to downgrade ripgrep

Just download the binary of an older release and put it somewhere in $PATH. You can even overwrite /usr/local/Cellar/ripgrep/*/bin/rg with that binary and brew pin that "version" until ?vim or vim-grepper is fixed.

[bluz71]

I was able to work-around the issue by finding a way to | head -1000 my results, to limit the number of lines saved into quickfix.

What was the syntax for you used to achieve that inside Vim using grepprg?

[sheldond]

@klas-genestack Thanks! Good points. If I end up comparing performance I will let you know.

@bluz71 I could not figure out how to do that purely in vim sadly (if you find it please let me know!), so I wrote a shell script to wrap rg, I use it like this:

A vimgrep script in path:

#!/bin/bash
rg --vimgrep --hidden --follow --max-columns=1000 --color=never --case-sensitive "$@" | head -n 1000 | sort

And in .vimrc:

if executable('vimgrep')
  set grepprg=vimgrep
elseif executable('rg')
  set grepprg=rg\ --vimgrep\ --hidden\ --follow\ --max-columns=1000\ --case-sensitive
endif
set grepformat=%f:%l:%c:%m,%f:%l:%m
command! -nargs=+ -complete=file QGrep execute 'silent grep! <args>' | copen
nnoremap gs/ :QGrep "" .<Left><Left><Left>

[jdsutherland]

I don't mean to be rude, but there's a lot of off-topic discussion in this issue now. @blueyed already posted a trivial fix (https://github.com/mhinz/vim-grepper/issues/244#issuecomment-860755993), so the discussion around how to replace vim-grepper isn't really warranted.

It seems that vim-grepper should merge a fix (though it probably needs tweaked to support Windows) given that the Neovim PR didn't fix this and it doesn't seem that ripgrep will change. @blueyed thoughts?

[ddeville]

https://github.com/neovim/neovim/pull/14812 by itself is not fixing the issue since the new option it introduces defaults to the original value. I've sent a PR that I've verified actually fixes it here https://github.com/mhinz/vim-grepper/pull/250.