Concurrent walk optimization

[GaruGaru]

Summary

Use golang.org/x/sync/errgroup to perform concurrent directory traversal / listing.

Fixes #17

Impact

Please delete options that are not relevant.

• [x] Performance improvement for file listing

Testing

Test Configuration:

• Go version: 1.18
• Hardware: I7 6700HQ, 16Gb Memory DDR4, SSD
• Operating system: Linux 5.10.146-1-MANJARO x86_64 GNU/Linux

Benchmarks

FileTreeBuilder-8     2.65s ± 9%     2.31s ± 4%  -12.95%  (p=0.000 n=10+10)

name               old alloc/op   new alloc/op   delta
FileTreeBuilder-8     297MB ± 0%     297MB ± 0%     ~     (p=0.280 n=10+10)

name               old allocs/op  new allocs/op  delta
FileTreeBuilder-8     3.00M ± 0%     3.00M ± 0%   -0.00%  (p=0.007 n=10+10)

[GaruGaru]

Comments fixed, if we want more performance gains probably a lazy eval for the tree is needed, but I guess that it would require a major refactor to be archived.

From my benchmarks the vast majority of CPU time is spent on system calls, in particular: Readdirsnames and os.Lstat.

[ozansz]

Comments fixed, if we want more performance gains probably a lazy eval for the tree is needed, but I guess that it would require a major refactor to be archived.

From my benchmarks the vast majority of CPU time is spent on system calls, in particular: Readdirsnames and os.Lstat.

I agree on the lazy evaluation. It would be a better approach but would definitely require a major update. Thanks for the remarks on Readdirsnames and os.Lstat too 👍 I will try to find if there’s a way to use less syscalls other than using lazy eval