search

jvirkki / dupd

CLI utility to find duplicate files
http://www.virkki.com/dupd
GNU General Public License v3.0
114 stars 16 forks source link

error: spurious unable to stat on smb network filesystem #34

Open tmsd2001 opened 4 years ago

tmsd2001 commented 4 years ago

Version 1.7

Scanning on SMB Network drives.

There are different variants of the error, with one file, dupd could stat the file with a second run. With another file that didn't help.

root@Ubuntu18:/mnt/backup3/omv_backup/omvbackup/usr# dupd scan --hidden -I Files: 4755 0 errors 12 s error: unable to stat /mnt/backup3/omv_backup/omvbackup/usr/sbin/cryptdisks_stop

root@Ubuntu18:/mnt/backup3/omv_backup/omvbackup/usr# dupd scan --hidden -I Files: 11880 0 errors 5248 ms error: unable to stat /mnt/backup3/omv_backup/omvbackup/usr/lib/python2.7/sitecustomize.py

root@Ubuntu18:/mnt/backup3/omv_backup/omvbackup/usr# dupd scan --hidden -I Files: 11832 0 errors 3247 m serror: unable to stat /mnt/backup3/omv_backup/omvbackup/usr/lib/python2.7/sitecustomize.py

root@Ubuntu18:/mnt/backup3/omv_backup/omvbackup/usr# dupd scan --hidden -I Files: 11986 0 errors 3497 ms error: unable to stat /mnt/backup3/omv_backup/omvbackup/usr/lib/python2.7/sitecustomize.p

dupd scan -v -v -v --hidden -I | head -10 output: Defaulting --path to [/mnt/backup3/omv_backup/omvbackup/usr] Will be using_fiemap (if available): 1 Reported RAM: 64250MB buffer limit: 32125MB Log level: INFO Claimed CPU cores: 20 Max open files: 1048566 Done initializing new database [/root/.dupd_sqlite] Set path_separator from db to  () Set scan_hidden from db to 1 database create time 1602745196213

jvirkki commented 4 years ago

'error: unable to stat' just means that the lstat() call returned an error. Not much dupd can do if that happens.

This can also happen if the file was deleted between the time scan saw it and tried to lstat() it. Do all these files exist?

I doubt there is a dupd bug here as it is directly a result of lstat() failing which is a filesystem issue.

tmsd2001 commented 4 years ago

sitecustomize.py As I have seen, it is a softlink., but the linked File is available. can also exclude softlinks? or can dupd skip the file, at the moment the program stops.

jvirkki commented 4 years ago

I'm not familiar with windows/smb, so not sure if it handles symlinks correctly.

Symbolic links are always excluded by dupd, there is no option to enable that (and will not add one). But the problem here is the lstat() call is failing. We don't know that this file is a symlink yet, before making the lstat() call. So there is nothing dupd can do to prevent the smb filesystem flakyness here.

It might be possible to just skip the file if this happens, I'll need to investigate.