search

sahib / rmlint

Extremely fast tool to remove duplicates and other lint from your filesystem
http://rmlint.rtfd.org
GNU General Public License v3.0
1.91k stars 132 forks source link

Output files terminated randomly #453

Closed nstephenh closed 3 years ago

nstephenh commented 3 years ago

Hello! I have a large number of files that need de-duplicated, and so I ran rmlint on my files, but when I ran the removal script I noticed it terminated due to a missing quote. I've inspected both the script and json file, and they both seem to stop mid-line. Any ideas what could be causing this?

The dataset is quite large (22TB) and it found nearly 4tb of dupes over 5 days 10 hours. I'm also accessing it via smb and storing the rmlint files to the smb share as well.

Thanks

sahib commented 3 years ago

Hello @nstephenh,

hard to tell without knowing what version of rmlint you were using. If you're using the last version:

The dataset is quite large (22TB) and it found nearly 4tb of dupes over 5 days 10 hours. I'm also accessing it via smb and storing the rmlint files to the smb share as well.

Just as a note: Scanning / removing over SMB will be very slow.

nstephenh commented 3 years ago

I'm going to close this because it likely was something related to my connection to my share. I re-ran rmlint from a local directory (still scanning the SMB share) so the script and json files were saved to my ssd. It went a lot faster and there were no issues. Besides this, I haven't seen any other issues with my SMB share though.

EDIT: nsh@nsh-Lenovo-ideapad-Y700-15ISK:~$ rmlint --version version 2.9.0 compiled: Aug 20 2020 at [19:43:54] "Odd Olm" (rev 2.1) compiled with: +mounts +nonstripped +fiemap +sha512 +bigfiles +intl +replay +xattr +btrfs-support

rmlint was written by Christopher Pahl and Daniel Thomas. The code at https://github.com/sahib/rmlint is licensed under the terms of the GPLv3.