Open billzt opened 6 years ago
I have an optimized fork of an old version which features multi process and a C extension and is expected to be slightly faster. You can try it at https://github.com/ylxdzsw/dedup . run make
before running to utilize the C extension.
Thanks, I'll have a try
The python script is too slow (several hours) and use high memeory (~70G)