Symlinks from wine result in walker infinite loop

harrypotterIUP commented 1 week ago

Checklist

[X] I have read through the manual page (man fzf)
[X] I have searched through the existing issues
[X] For bug reports, I have checked if the bug is reproducible in the latest version of fzf

Output of `fzf --version`

0.55.0 (fc69308)

OS

[X] Linux
[ ] macOS
[ ] Windows
[ ] Etc.

Shell

[X] bash
[ ] zsh
[ ] fish

Problem / Steps to reproduce

A simple call fzf in my home folder takes up all CPU and memory resources and seems to run forever. Initially, I assumed that my laptop is just too slow for fzf (see #4014), but now it looks more like a problem with an infinite loop of the fzf walker.

I noticed that the issue disappears with fzf --walker=file,hidden (omitting follow). A subsequent investigation of all relevant subfolders which contain symlinks (with find . -type l | sed 's/.\/\([^/]\+\).*/\1/p' | uniq) showed a lot of wine prefixes.

This suspicion indeed turned out to be the issue: The problem disappears with fzf --walker-skip=drive_c,dosdevices (two automatically created folders/symlinks in any wine prefix) and the walker finishes very quickly.

As expected, the issue also disappears when disabling the internal walker and using FZF_DEFAULT_COMMAND='find .' fzf instead. When making find follow symlinks find -L . the command runs very long and there are lots of "Too many levels of symbolic links" and "File system loop detected" errors.

I am a little surprised, though, that I should be the first one to report this issue, since wine is a widespread program? And the symlinks from wine won't be the only case that lead to infinite loops I guess? Interested to hear if others can reproduce the issue and how to deal with symlinks in general. :+1:

LangLangBart commented 1 week ago

Interested to hear if others can reproduce the issue and how to deal with symlinks in general.

I reproduced the issue with fnm[^1] where enabling its shell setup in my zshrc causes a new symlink directory to be created for every shell instance, quickly resulting in thousands of symlinks under ~/.local/state/fnm_multishells.

The fnm's maintainer provided more details here:

https://github.com/Schniz/fnm/issues/1157#issuecomment-2143788670

Tools like fzf, rg, and fd take significantly longer when following symlinks in these directories.

cd ~/.local/state/fnm_multishells
find . -type l -depth 1 | wc -l
    4608

# fzf version: 0.55.0
unset FZF_DEFAULT_COMMAND
time command fzf --walker=file,follow --filter ''
user=86.46s system=241.46s cpu=232% total=2:21.14

# fd version: 10.2.0
time command fd --follow --color=never
user=163.76s system=582.79s cpu=496% total=2:30.25

# ripgrep version: 14.1.0
time command rg --follow --files --color=never
user=298.48s system=1245.37s cpu=614% total=4:11.40

Hence, I would suggest this isn't a bug but a design choice by fzf to enable follow by default. Users should be aware and use options like fzf's --walker-skip= to avoid unnecessary directories.

[^1]: GitHub - Schniz/fnm: 🚀 Fast and simple Node.js version manager, built in Rust

harrypotterIUP commented 1 week ago

Thank you very much for looking into this!

Hence, I would suggest this isn't a bug but a design choice by fzf to enable follow by default

Yes, I agree. In previous versions, find -L was also used. So whoever has issues with symlinks with the walker now, already had them before with find -L, too. No need to change the default follow.

Nevertheless, I think there are still two points worth addressing:

Is it possible to make the walker smarter to recognize symlink infinite loops? In other words, there would be no need to deactivate follow if there's a way to avoid that heavy overload caused by looping/repeatedly following the same symlinks.
Is it possible to make --walker-skip smarter? For example, one of the symlinks created by wine is wine/drive_c/users/<username>/Downloads. Setting --walker-skip=drive_c is excluding the whole wine file structure (excluding program files, too) which very well might contain files I want fzf to match, yet setting --walker-skip=Downloads might be even worse if my actual downloads folder shares the same name. So maybe a feature like --walker-skip=drive_c/users or even --walker-skip=drive_c/users/*/Downloads for a more precise directory name blacklist could help?

junegunn / fzf