idrassi
commented
1 month ago Thank you for reporting these issues.
For the first point, the issue is not caused by a handle leak in the implementation. The core issue lies in the fact that, although each file handle is eventually closed, the rate of handle creation by the main thread is significantly faster than the rate of handle closure by the worker threads. This can lead to a temporary buildup of open handles, especially when processing a large number of files. To resolve this, I will implement a change to close the handle in the main thread as soon as the hash job is queued in the thread pool, and then have the worker thread reopen the file when its processing begins.
For the second point, I agree that documenting this side effect of using -thread on mechanical disks is warranted. I'm also considering adding a detection mechanism in DirHash to identify such drives, so that a warning can be displayed when -threads is used.
For the third point, it is definitely a good idea to sort entries in the output SUM file. Currently, we write the result of hashing to the SUM file as they arrive from the thread pool, so the order is not guaranteed. To add ordering, there are two approaches:
- Sort the contents of the SUM file after we have finished hashing all files. This would keep the existing code as it is and apply a post-processing step to the output file.
- Keep the result of hashing in memory and avoid writing anything to the output file until all hashing is done. We would then sort the entries in memory and finally write them sorted to the output file.
I think I will go with the post-processing approach since it has a lower memory footprint and is simpler to implement.
I will let you know when I implement these changes.
groone
When verifying blake3 hashsums of a large deeply nested directory with -threads parameter DirHash keeps open a very large number of file handles. In my use case I saw it keeping open over 9000 file handles. That caused out of memory error in the virtual drive of a popular cloud sync friendly encryption software "Cryptomator".
And second point: Documentation should be updated to note that -threads parameter can significantly slow down the verification if files are on a spinning disk. After trying it out I learned that my use case takes 6 hours of disk thrashing with -threads and 1.5 hours without it. Spinning disks should be accessed in sequential order instead.
Third point: When creating hashsum file of large deeply nested directory with -threads parameter, the file names are not in lexicographical order and diffs between hashsum files are not usable