markfasheh / duperemove

Tools for deduping file systems
GNU General Public License v2.0
794 stars 78 forks source link

duperemove: increase open file limit from default to maximum available #269

Closed trofi closed 3 years ago

trofi commented 3 years ago

On 32-CPU machine with SSD I occasionally get -EMFILE errors when deduping as:

duperemove -dr --hashfile=dupes.db /

My guess is that we exhausl relatively small 1K default RLIMIT_NOFILE. The change cranks up the limit from 1K to 512K on x86_64 linux.