[BUG] fd doesn't detect symbolic link loops, runs forever

[skierpage]

Checks

• [X] I have read the troubleshooting section and still think this is a bug.

Describe the bug you encountered:

I wanted to find "control knobs" for the fan in my Thinkpad laptop running Linux. So

% fd --follow fan /sys/class/hwmon/hwmon3 | tee /tmp/fd_endless_find_hwmon3.out

runs for several seconds with no output, then finds 14 matching files, then runs for more seconds... and never completes.

/sys/class is notorious for being symbolic link heavy. But find immediately detects and warns about symbolic link forests, while running forever (?):

find -L /sys/class/hwmon/hwmon3 -name '*fan*'
find: File system loop detected; ‘/sys/class/hwmon/hwmon3/device/device/wakeup/wakeup19/device’ is part of the same file system loop as ‘/sys/class/hwmon/hwmon3/device/device’.
find: File system loop detected; ‘/sys/class/hwmon/hwmon3/device/device/wakeup/wakeup19/subsystem/wakeup4/device/wakeup/wakeup4’ is part of the same file system loop as ‘/sys/class/hwmon/hwmon3/device/device/wakeup/wakeup19/subsystem/wakeup4’.

etc. Meanwhile recursive ls takes a long time to complete, but generates lots of warnings as it goes:

$ ls --dereference -lR /sys/class/hwmon/hwmon3
ls: '/sys/class/hwmon/hwmon3/device/device/driver/PNP0C0A:00/driver': not listing already-listed directory
ls: '/sys/class/hwmon/hwmon3/device/device/driver/PNP0C0A:00/physical_node/firmware_node': not listing already-listed directory
...

Describe what you expected to happen:

Either behavior (warn about file system loops and/or don't re-traverse directories already descended) is better than running forever with no indication of any problem.

I'm not sure how to create a reproducible test case. I guess I could upload an attachment that's a tar file of the forest of symlinks if it would help. Thanks for a great tool :+1: .

What version of fd are you using?

8.7.0

Which operating system / distribution are you on?

Linux 6.3.8-200.fc38.x86_64 x86_64

[xweskingx]

If you run the command with the --show-errors flag, it will report the file system loop and the too many levels of symbolic links.

fd --follow --show-errors fan /sys/class/hwmon/hwmon3

[fd error]: File system loop found: <Path> [fd error]: <Path> : Too many levels of symbolic links (os error 40)

Looking at the walk implementation in the ignore crate, it does appear to properly handle those errors by not descending into the directory.

It looks like it would be possible to report both situations uniquely, although checking the too many levels error would require the #![feature(io_error_more)] feature.

[tmccombs]

FWIW, find -L -name '*fan*' /sys/class/hwmon/hwmon3 also struggles with this.

I think the problem might not just be that there cyclical links, but the structure is such that it takes a long time to discover them.

I think what happens is there is a symlink that points to an ancestor folder. So fd starts doing a depth-first traversal of that directory, and eventually hits the same symlink again, and follows that, over and over until it hits the depth limit. Once it hits that error it stops traversing, but it still keeps going through the depth-first traversal for all the higher levels, because it doesn't know which symlink in the change caused the problem. If we had something that checked if a symlink pointed to an ancestor of the symlink that might help, in some cases. But I'm not sure if that would be sufficient to deal with the labyrinth of symlinks in /sys.

In my experience, following symlinks in /sys and /proc with a recursive search like with fd, find, grep, etc. is generally not a great idea.

[tavianator]

I think the problem might not just be that there cyclical links, but the structure is such that it takes a long time to discover them.

Yeah, see https://github.com/sharkdp/fd/issues/378#issuecomment-452084068

[powerman]

Also happens in home directory, because of Steam:

$ ls -l ~/.local/share/Steam/steamapps/compatdata/0/pfx/dosdevices/z:
lrwxrwxrwx 1 powerman powerman 1 окт 17  2021 /home/powerman/.local/share/Steam/steamapps/compatdata/0/pfx/dosdevices/z: -> /

$ fd --show-errors --follow . ~/.local/share/Steam >/dev/null
[fd error]: File system loop found: /home/powerman/.local/share/Steam/steamapps/compatdata/0/pfx/dosdevices/z:/sys/block/zram0/bdi/subsystem/253:0 points to an ancestor /home/powerman/.local/share/Steam/steamapps/compatdata/0/pfx/dosdevices/z:/sys/block/zram0/bdi
[fd error]: File system loop found: /home/powerman/.local/share/Steam/steamapps/compatdata/0/pfx/dosdevices/z:/sys/block/zram0/bdi/subsystem/8:0/subsystem points to an ancestor /home/powerman/.local/share/Steam/steamapps/compatdata/0/pfx/dosdevices/z:/sys/block/zram0/bdi/subsystem
...