We are using the file-system IO components at Tuist, and I'd like to bring up something that we notice regarding the performance of the NIOFileSystem.FileSystem.removeItem function.
When we run it against a large directory with many nested directories and and files, we noticed the performance is much worse than using FileManager or just rm -rf in a Linux environment. We benchmarked the performance with hyperfine and the results are below.
Looking at the implementation, it seems that we are not parallelising at all, so I was wondering if this is something intentional, or if it's ok to refactor that code to run as much as possible in parallel.
Bechmarks
With NIOFileSystem
Benchmark
Time (mean ± σ): 67.939 s ± 4.450 s [User: 61.906 s, System: 3.205 s]
Range (min … max): 63.684 s … 77.757 s 10 runs
With Foundation's FileManager
Benchmark:
Time (mean ± σ): 5.857 s ± 0.251 s [User: 1.430 s, System: 2.067 s]
Range (min … max): 5.419 s … 6.190 s 10 runs
With rm -rf
Benchmark:
Time (mean ± σ): 3.150 s ± 0.386 s [User: 0.022 s, System: 1.666 s]
Range (min … max): 2.225 s … 3.585 s 10 runs
Thanks for filling this! This is definitely something that we would take a patch for and parallelise the removal similar to how we did for copying in https://github.com/apple/swift-nio/pull/2806.
👋🏼 Hi,
First of all, thanks for this amazing work.
We are using the file-system IO components at Tuist, and I'd like to bring up something that we notice regarding the performance of the
NIOFileSystem.FileSystem.removeItem
function.When we run it against a large directory with many nested directories and and files, we noticed the performance is much worse than using
FileManager
or justrm -rf
in a Linux environment. We benchmarked the performance with hyperfine and the results are below.Looking at the implementation, it seems that we are not parallelising at all, so I was wondering if this is something intentional, or if it's ok to refactor that code to run as much as possible in parallel.
Bechmarks
With NIOFileSystem
Benchmark
With Foundation's FileManager
Benchmark:
With
rm -rf
Benchmark: Time (mean ± σ): 3.150 s ± 0.386 s [User: 0.022 s, System: 1.666 s] Range (min … max): 2.225 s … 3.585 s 10 runs