search

lsd-rs / lsd

The next gen ls command
Apache License 2.0
13.59k stars 438 forks source link

LSD is very slow compared to ls when listing large directories #378

Closed JamsNJellies closed 6 days ago

JamsNJellies commented 4 years ago

Listing the Void Linux repo srcpkgs directory (about 12055 directories) takes far longer with lsd than it does with GNU ls.

lsd --color never 1.39s user 0.12s system 99% cpu 1.521 total lsd 1.42s user 0.12s system 98% cpu 1.559 total ls 0.01s user 0.02s system 91% cpu 0.034 total

ftpd commented 3 years ago

+1 to this. I've just did the other test: compared lsd and exa when listing long output and tree view: exa -l -T 2.54s user 6.14s system 47% cpu 18.298 total lsd -l --tree 332.53s user 96.61s system 50% cpu 14:13.04 total (obviously no info from vanilla ls, as it doesn't have tree functionality).

I did it on my 13GB home directory (macOS, so many small files in ~/Library).

ben2talk commented 3 years ago

Yes. Also using in .zshrc or .bashrc

Automatically ls when changing directory

cd() { builtin cd "$@" && lsd

lsd comes up with Permission denied (oserror 13). errors when opening Dolphin terminal...

Compare with...

# Automatically ls when changing directory
cd() {
  builtin cd "$@" && exa --icons --group-directories-first  
}
meain commented 3 years ago

@ben2talk Permission denied is probably a different issues: https://github.com/Peltoche/lsd/issues/79

hedonihilist commented 3 years ago

Same here. The time of listing 287188 files:

❯ ls | wc -l     
287188
❯ time lsd 1>/dev/null
lsd > /dev/null  6.50s user 2.60s system 99% cpu 9.131 total
❯ time ls 1>/dev/null
ls > /dev/null  0.42s user 0.10s system 99% cpu 0.525 total

Whyt lsd is much slower than ls? Is there anything we can do?

zwpaper commented 3 years ago

there is a PR for speed up, but not yet finished. https://github.com/Peltoche/lsd/pull/441.

arctic-penguin commented 1 year ago

Since #441 was closed, is there any indication whether this issue will be fixed?

arctic-penguin commented 1 year ago

Here is how lsd compares to exa and ls at the moment. Test cases is a directory with 10,376 files (a fuzzing corpus).

$ hyperfine ls exa lsd
Benchmark 1: ls
  Time (mean ± σ):      16.6 ms ±   1.9 ms    [User: 12.4 ms, System: 4.1 ms]
  Range (min … max):    13.1 ms …  20.3 ms    144 runs

Benchmark 2: exa
  Time (mean ± σ):      47.9 ms ±   5.7 ms    [User: 22.5 ms, System: 25.0 ms]
  Range (min … max):    38.8 ms …  61.5 ms    57 runs

Benchmark 3: lsd
  Time (mean ± σ):      7.213 s ±  0.310 s    [User: 0.897 s, System: 1.699 s]
  Range (min … max):    7.043 s …  8.060 s    10 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  'ls' ran
    2.89 ± 0.47 times faster than 'exa'
  434.83 ± 52.45 times faster than 'lsd'

So plain-old ls is 400 times faster than lsd.

$ ls --version
ls (GNU coreutils) 9.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Richard M. Stallman and David MacKenzie.
$ lsd --version
lsd 0.23.1
$ exa --version
exa - list files on the command-line
v0.10.1 [+git]
https://the.exa.website/
planet36 commented 1 year ago

Use strace with ls, exa, and lsd in a large directory, and notice that lsd is making many more system calls that the other 2.

strace ls --color=never -l --sort=none 2> ~/strace.ls.txt strace exa --color=never --no-icons --long --sort=none 2> ~/strace.exa.txt strace lsd --color=never --icon=never --long --sort=none 2> ~/strace.lsd.txt

wc --lines --total=never strace.*.txt

3183 strace.exa.txt 4900 strace.ls.txt 408402 strace.lsd.txt

planet36 commented 1 year ago

You can compare the number of particular system calls with these commands.

sed -E -e 's/\(.*//g' ~/strace.ls.txt | sort | uniq -c sed -E -e 's/\(.*//g' ~/strace.exa.txt | sort | uniq -c sed -E -e 's/\(.*//g' ~/strace.lsd.txt | sort | uniq -c

planet36 commented 1 year ago

This is how I looked into the issue of lsd's bad performance.

I created a directory in /tmp with 10,000 empty files. On my Linux system, /tmp is a tmpfs, and bash was the shell I used.

mkdir /tmp/1E4
cd /tmp/1E4
touch $(seq 1E4)

Then I compared the execution time of exa, lsd, ls, and uu-ls on that directory.

The options used were intended to disable features that differ between programs (such as icons and colors) but still display "long" output.

printf '\n\nexa'   ; time LC_ALL=C exa --color=never --no-icons --long --sort=none > /dev/null
printf '\n\nlsd'   ; time LC_ALL=C lsd --color=never --icon=never --long --sort=none --ignore-config > /dev/null
printf '\n\nls'    ; time LC_ALL=C ls --color=never -l --sort=none > /dev/null
printf '\n\nuu-ls' ; time LC_ALL=C uu-ls --color=never -l --sort=none > /dev/null
exa
real    0m0.082s
user    0m0.064s
sys     0m0.045s

lsd
real    0m4.467s
user    0m1.506s
sys     0m1.913s

ls
real    0m0.028s
user    0m0.010s
sys     0m0.016s

uu-ls
real    0m0.036s
user    0m0.025s
sys     0m0.010s

Next I logged the system calls made by each program.

LC_ALL=C strace exa --color=never --no-icons --long --sort=none > /dev/null 2> ~/strace.exa.txt
LC_ALL=C strace lsd --color=never --icon=never --long --sort=none --ignore-config > /dev/null 2> ~/strace.lsd.txt
LC_ALL=C strace ls --color=never -l --sort=none > /dev/null 2> ~/strace.ls.txt
LC_ALL=C strace uu-ls --color=never -l --sort=none > /dev/null 2> ~/strace.uu-ls.txt
cd
wc --lines --total=never strace.*.txt
    21320 strace.exa.txt
  2777707 strace.lsd.txt
    20512 strace.ls.txt
    10499 strace.uu-ls.txt

You can see that lsd had many more system calls than the others. It's strace output was about 176M, so keep that in mind if you test on a directory with more files.

Here's a count of how many times each system call was made for each program.

printf '\n\nexa\n'   ; sed -n -E -e 's/\(.*//gp' strace.exa.txt   | sort | uniq -c
printf '\n\nlsd\n'   ; sed -n -E -e 's/\(.*//gp' strace.lsd.txt   | sort | uniq -c
printf '\n\nls\n'    ; sed -n -E -e 's/\(.*//gp' strace.ls.txt    | sort | uniq -c
printf '\n\nuu-ls\n' ; sed -n -E -e 's/\(.*//gp' strace.uu-ls.txt | sort | uniq -c
exa
      1 access
      2 arch_prctl
     34 brk
      4 clone3
     17 close
      1 execve
      1 exit_group
   1059 futex
     11 getdents64
      2 getrandom
      1 ioctl
     56 mmap
     19 mprotect
      5 mremap
      5 munmap
     16 newfstatat
     19 openat
      1 poll
      2 pread64
      2 prlimit64
     36 read
      1 rseq
      6 rt_sigaction
      9 rt_sigprocmask
      2 sched_getaffinity
      1 set_robust_list
      1 set_tid_address
      3 sigaltstack
  10001 statx
  10001 write

lsd
      1 access
      2 arch_prctl
  34008 brk
 800395 close
  20306 connect
  20002 epoll_create1
 120012 epoll_ctl
  80008 epoll_pwait2
    989 epoll_wait
      1 execve
      1 exit_group
      2 futex
  40013 getdents64
      1 getpid
      3 getrandom
      2 ioctl
  30003 lgetxattr
  30003 lseek
     30 mmap
      8 mprotect
      9 mremap
      7 munmap
 470059 newfstatat
 940106 openat
      1 poll
      6 prctl
      2 pread64
      2 prlimit64
  30015 read
  10001 readlink
  20991 recvfrom
      1 rseq
      5 rt_sigaction
  40004 rt_sigprocmask
      1 sched_getaffinity
  20002 sendto
      1 set_robust_list
      1 set_tid_address
      3 sigaltstack
  20306 socket
  10001 statx
  20002 timerfd_create
  20389 timerfd_settime
      1 write

ls
      1 access
      2 arch_prctl
      5 brk
     96 close
      6 connect
      2 epoll_create1
     12 epoll_ctl
      8 epoll_pwait2
      1 execve
      1 exit_group
      2 futex
     13 getdents64
      1 getpid
      2 getrandom
  10000 getxattr
      2 ioctl
      3 lseek
     28 mmap
      7 mprotect
      4 mremap
      3 munmap
     59 newfstatat
    105 openat
      6 prctl
      2 pread64
      1 prlimit64
     12 read
      2 recvfrom
      1 rseq
      4 rt_sigprocmask
      2 sendto
      1 set_robust_list
      1 set_tid_address
      6 socket
  10000 statx
      2 timerfd_create
      3 timerfd_settime
    105 write

uu-ls
      1 access
      2 arch_prctl
     11 brk
     95 close
      6 connect
      2 epoll_create1
     12 epoll_ctl
      8 epoll_pwait2
      1 execve
      1 exit_group
      2 futex
     13 getdents64
      1 getpid
      3 getrandom
      2 ioctl
      3 lseek
     28 mmap
      8 mprotect
      4 mremap
      5 munmap
     60 newfstatat
    108 openat
      1 poll
      6 prctl
      2 pread64
      2 prlimit64
     15 read
      2 recvfrom
      1 rseq
      5 rt_sigaction
      4 rt_sigprocmask
      1 sched_getaffinity
      2 sendto
      1 set_robust_list
      1 set_tid_address
      3 sigaltstack
      6 socket
  10001 statx
      2 timerfd_create
      2 timerfd_settime
     65 write

I used a spreadsheet to put the strace results in a table for easier comparison.

System call exa lsd ls uu-ls
access 1 1 1 1
arch_prctl 2 2 2 2
brk 34 34008 5 11
clone3 4
close 17 800395 96 95
connect 20306 6 6
epoll_create1 20002 2 2
epoll_ctl 120012 12 12
epoll_pwait2 80008 8 8
epoll_wait 989
execve 1 1 1 1
exit_group 1 1 1 1
futex 1059 2 2 2
getdents64 11 40013 13 13
getpid 1 1 1
getrandom 2 3 2 3
getxattr 10000
ioctl 1 2 2 2
lgetxattr 30003
lseek 30003 3 3
mmap 56 30 28 28
mprotect 19 8 7 8
mremap 5 9 4 4
munmap 5 7 3 5
newfstatat 16 470059 59 60
openat 19 940106 105 108
poll 1 1 6 1
prctl 6 6
pread64 2 2 2 2
prlimit64 2 2 1 2
read 36 30015 12 15
readlink 10001
recvfrom 20991 2 2
rseq 1 1 1 1
rt_sigaction 6 5 5
rt_sigprocmask 9 40004 4 4
sched_getaffinity 2 1 1
sendto 20002 2 2
set_robust_list 1 1 1 1
set_tid_address 1 1 1 1
sigaltstack 3 3 3
socket 20306 6 6
statx 10001 10001 10000 10001
timerfd_create 20002 2 2
timerfd_settime 20389 3 2
write 10001 1 105 65

Here are some of the calls that stood out to me.

grep -F 'readlink(' strace.lsd.txt

grep -E '^(socket|connect)\(' strace.lsd.txt | sort | uniq -c | sort -n | tail

grep -F '/etc/nsswitch.conf' strace.lsd.txt | sort | uniq -c | sort -n | tail

grep -E '^openat\(.+"/' strace.lsd.txt | sort | uniq -c | sort -n | tail

tl;dr: WTH?!

CamJN commented 1 year ago

as a workaround for now i turned off total-size which resulted in a 3632x speedup.

planet36 commented 1 year ago

as a workaround for now i turned off total-size which resulted in a 3632x speedup.

The --total-size option isn't enabled by default. Not giving that option doesn't fix the systemic performance problems of lsd that I've shown above.

CamJN commented 1 year ago

It was enabled in my config file, I probably turned it on at some point, but it seems pretty much mandatory to have it disabled. Like I said the difference was massive, even if it doesn't address the overall slowness.

planet36 commented 1 year ago

I added total-size: false to my config.yaml, but I didn't see a difference in performance.

Commands:

mkdir -p /tmp/1E4 ; cd /tmp/1E4 ; touch $(seq 1E4)
printf '\neza'          ; time LC_ALL=C command eza   --color=never --sort=none -l --no-icons --group > /dev/null
printf '\nlsd'          ; time LC_ALL=C command lsd   --color=never --sort=none -l --icon=never --ignore-config > /dev/null
printf '\nlsd (config)' ; time LC_ALL=C command lsd   --color=never --sort=none -l --icon=never > /dev/null
printf '\nls'           ; time LC_ALL=C command ls    --color=never --sort=none -l > /dev/null
printf '\nuu-ls'        ; time LC_ALL=C command uu-ls --color=never --sort=none -l > /dev/null

Output:

eza real 0m0.093s user 0m0.060s sys 0m0.072s

lsd real 0m5.200s user 0m1.325s sys 0m1.688s

lsd (config) real 0m5.974s user 0m1.165s sys 0m1.659s

ls real 0m0.064s user 0m0.034s sys 0m0.029s

uu-ls real 0m0.056s user 0m0.041s sys 0m0.014s

tjex commented 1 year ago

I just set total-size: false and it performs fast and snappy again. Maybe it's a clearer test to remove --ignore-config from the lsd command?

I added total-size: false to my config.yaml, but I didn't see a difference in performance.

Commands:

printf '\nlsd'          ; time LC_ALL=C command lsd   --color=never --sort=none -l --icon=never --ignore-config > /dev/null
planet36 commented 1 year ago

I just set total-size: false and it performs fast and snappy again. Maybe it's a clearer test to remove --ignore-config from the lsd command?

No. In my example there were two calls of lsd: one that uses the config file, and one that ignores it. The intent was to show that neither method has acceptable performance.

tjex commented 1 year ago

ahhhhh. missed that (obviously).. Sorry!

luketych commented 1 month ago

Any updates on this?

Yes it is very very slow to the point of being unusable. Please fix, it's not that difficult. I can share my code for a custom tree command that prints in real-time and executes in less than 1 second.

zwpaper commented 6 days ago

this should be fixed by v1.1.5, at least for first step:

bash-5.1$ lsd --version
lsd 1.1.5
bash-5.1$ printf '\n\nlsd'   ; time LC_ALL=C lsd   --color=never --sort=none -1 --icon=never --ignore-config > /dev/null

lsd
real    0m1.925s
user    0m0.328s
sys 0m1.588s

I'd like to close this issue to mark this update, in case of misleading to the slow performance.

feel free to create a new issue if any other performance issues are met.

I will keep improving the performance on lsd.