search

pkolaczk / fclones

Efficient Duplicate File Finder
MIT License
1.94k stars 75 forks source link

Obey the open file descriptor limit when grouping files #200

Closed th1000s closed 1 year ago

th1000s commented 1 year ago

Given enough CPU cores and the multiplication factor fclones uses, the number of threads and therefore approximately the maximum number of open files can exceed the usual limit of 1024 on Linux. On macOS this tends to be even lower.

Then some of the "fetch file extents mapping" and "compute hash of file" steps fail with "Too many open files (os error 24)".

To fix this, retrieve the current limit, increase it if possible and stay below it.

Only used on Unix systems.

th1000s commented 1 year ago

This is set globally and not in main(), so this can't be changed when using fclones as a library.

Two local variables, in group/file, could also be used before threads are created, but then the counter would have to be passed into various functions.