search

sharkdp / fd

A simple, fast and user-friendly alternative to 'find'
Apache License 2.0
32.6k stars 785 forks source link

Can fd avoid symlink cycles? #938

Open timkgh opened 2 years ago

timkgh commented 2 years ago

It seems easy to get fd to follow symlinks and get it stuck in a cycle that's hard to detect to add the right exclude. E.g. Wine on Linux creates a symlink ~/.wine/dosdevices/z: --> / I was looking for a config file that I knew was somewhere in my home dir in one of the hidden dirs, so I ran fd -H -L ... that got stuck with high CPU use.

tmccombs commented 2 years ago

Are you sure it's a cycle, and it's not just slow because it is searching all of the root filesystem (/)?

fd does actually detect cycles in symlinks, at least in some cases. See https://github.com/sharkdp/fd/issues/180#issuecomment-347633820.

However, searching all of / will almost certainly take quite a bit longer than searching ~, unless your home directory has the majority of your files.

Probably what is really needed here, is a way to filter based on the resolved path of symlinks, but that is kind of niche use case, and certainly a somewhat subtle option to use.

Also note that if you don't include the -L option then fd won't follow symlinks.

timkgh commented 2 years ago

I understand what -L does, but as mentioned above I was looking for a config file that is somewhere in a maze of symlinks so I do want to follow links.

tavianator commented 2 years ago

If you search / with -L, it will eventually find /sys and then you will experience https://github.com/sharkdp/fd/issues/378. That bug is "fixed" in the sense that memory no longer grows unboundedly, but it will still take a long time unless you limit with --max-depth for the reason I detailed in https://github.com/sharkdp/fd/issues/378#issuecomment-452084068.

timkgh commented 2 years ago

I see, that's probably what's happening then and not cycles. Could fd warn and stop when it hits some of these well known locations as default behavior and only force it to traverse everything with a flag? It's not obvious what is happening when using fd.

tavianator commented 2 years ago

One thing that might help is --xdev.

I don't think stopping when it encounters /sys is workable, the user might want to search /sys. It might make sense to warn when descending into /sys with -L, but it's only one example of the potential problem.

Norodix commented 2 years ago

I encountered a similar problem. Wine's z: directory that links to / is quite common, I would not describe it as a niche use case. I use fd through fzf currently, so my workaround is to add --exclude "z:/" to all configured fzf commands, but it is not very feasible to add this to every command when using fd by itself.

I think it would be nice to have an option to follow symlinks only if it does not point outside of the search's root directory. That way when searching in $HOME it would not try to traverse the full filesystem for each wine prefix.

tavianator commented 2 years ago

I think it would be nice to have an option to follow symlinks only if it does not point outside of the search's root directory. That way when searching in $HOME it would not try to traverse the full filesystem for each wine prefix.

This is a neat idea, basically RESOLVE_BENEATH from LInux's new openat2() syscall. But it would be fairly difficult to implement on other platforms. I guess cap-std already implements it.