pkolaczk / fclones

Efficient Duplicate File Finder
MIT License
1.87k stars 70 forks source link

Windows-Test : OS-Error 87 #143

Closed Mae72DE closed 2 years ago

Mae72DE commented 2 years ago

[2022-08-03 08:34:01.175] fclones.exe: warn: Failed to read file identifier of 'C:\adbLink\remotes\keymap1.xml': Falscher Parameter. (os error 87)

It seems on every file I got the OS-Error 87 Thats all I can report - sorry.

64-Bit-Windows7 (PC-Perfomance-Index: 5,6) "DFHL.exe" (Duplicate File Hard Linker/(https://www.jensscheffler.de/dfhl) needs more than 3 Days. "fclones.exe" less than 3 hours - but … it don't really works … Test of "finddupe.exe" (v1.25 compiled May 11 2017) ist just running …

pkolaczk commented 2 years ago

I cannot reproduce it on Windows 10, nor on Wine. Unfortunately I have no access to a Windows 7 machine.

The code that's throwing this error is the code for obtaining a file identifier for given file:

  #[cfg(windows)]
    pub fn new(file: &Path) -> io::Result<FileId> {
        Self::from_file(&fs::File::open(file.to_path_buf())?).map_err(|_| {
            io::Error::new(
                ErrorKind::Other,
                format!(
                    "Failed to read file identifier of {}: {}",
                    file.display(),
                    io::Error::last_os_error()
                ),
            )
        })
    }

    #[cfg(windows)]
    pub fn from_file(file: &fs::File) -> io::Result<FileId> {
        use std::os::windows::io::*;
        use winapi::ctypes::c_void;
        use winapi::um::fileapi::FILE_ID_INFO;
        use winapi::um::minwinbase::FileIdInfo;
        use winapi::um::winbase::GetFileInformationByHandleEx;
        let handle = file.as_raw_handle();
        unsafe {
            let mut file_id: FILE_ID_INFO = std::mem::zeroed();
            let file_id_ptr = (&mut file_id) as *mut _ as *mut c_void;
            const FILE_ID_SIZE: u32 = std::mem::size_of::<FILE_ID_INFO>() as u32;
            match GetFileInformationByHandleEx(handle, FileIdInfo, file_id_ptr, FILE_ID_SIZE) {
                0 => Err(io::Error::new(
                    ErrorKind::Other,
                    format!(
                        "Failed to read file identifier: {}",
                        io::Error::last_os_error()
                    ),
                )),
                _ => Ok(FileId {
                    device: file_id.VolumeSerialNumber as u64,
                    inode: u128::from_be_bytes(file_id.FileId.Identifier),
                }),
            }
        }
    }

Quite likely the GetFileInformationByHandleEx call fails on Windows 7.

pkolaczk commented 2 years ago

Aaargh: https://stackoverflow.com/questions/70547710/alternative-to-getfileinformationbyhandleex-fileidinfo-for-windows-7 Looks like this call is not supported on Windows 7 :(

Fortunately, there is an older API I could use as a fallback. Fix coming...

pkolaczk commented 2 years ago

Does this one work for you?

fclones.zip

Mae72DE commented 2 years ago

…> fclones.exe group -o Test.Txt "C:\" [2022-08-03 19:49:20.640] fclones.exe: error: Can't access 'C:"': Die Syntax für den Dateinamen, Verzeichnisnamen oder d ie Datenträgerbezeichnung ist falsch. (os error 123) [2022-08-03 19:49:20.641] fclones.exe: error: Some input paths could not be accessed.

a Windows-User like me is like to use Dopple-Quotes for strings in command-Line, but without:

…> fclones.exe group -o Test.Txt C:\ [2022-08-03 19:52:20.071] fclones.exe: info: Started grouping [2022-08-03 19:52:20.080] fclones.exe: warn: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen P rozess verwendet wird. (os error 32) [2022-08-03 19:52:20.424] fclones.exe: warn: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen P rozess verwendet wird. (os error 32) [2022-08-03 19:52:22.842] fclones.exe: warn: Failed to read dir 'C:\$RECYCLE.BIN\S-1-5-20': Zugriff verweigert (os erro r 5) 1/6: Scanning files [ <===> ] 271

Its seems to work!!! 1/6: Scanning files [ <===> ] 3484

Now I've to wait, "FindDupe" need 8:21 h:m to process all 1034550 files (739527571 kBytes) or KB? But if 'fclones' is half as fast as the programmer we don't have to wait long ;-)

pkolaczk commented 2 years ago

Glad to hear that!

BTW: I noticed that some antivirus software had a huge impact on performance of fclones on windows. Disabling Windows Defender made a huge (10x or more) difference for me.

Mae72DE commented 2 years ago

… to complete the Feedback:

[2022-08-03 23:32:30.675] fclones.exe: info: Scanned 1131827 file entries [2022-08-03 23:32:31.225] fclones.exe: info: Found 1002054 (750.9 GB) files matching selection criteria [2022-08-03 23:32:31.778] fclones.exe: info: Found 819273 (152.2 GB) candidates after grouping by size [2022-08-03 23:32:31.927] fclones.exe: info: Found 819273 (152.2 GB) candidates after grouping by paths [2022-08-04 04:52:08.584] fclones.exe: info: Found 212794 (124.3 GB) candidates after grouping by prefix [2022-08-04 04:52:53.057] fclones.exe: info: Found 212794 (124.3 GB) candidates after grouping by suffix [2022-08-04 06:30:23.920] fclones.exe: info: Found 203165 (120.1 GB) redundant files

• Some Files produce an Error:

[2022-08-03 21:56:48.243] fclones.exe: warn: Failed to read dir 'C:\ProgramData\Microsoft\Crypto\PCPKSP\WindowsAIK': Zu griff verweigert (os error 5)

• But some Errors without an File-Name:

[2022-08-03 21:56:48.240] fclones.exe: warn: Zugriff verweigert (os error 5)

I miss the Info of how many files not readable, that may be:

[2022-08-03 22:52:55.268] fclones.exe: warn: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen P rozess verwendet wird. (os error 32)

It would be nice to have a counter for that, too.

pkolaczk commented 2 years ago

Just out of curiosity - what type of hard drive were you using? A local HDD? Network drive? What RPM? What filesystem? Fclones on Windows doesnt do the seek time optimization as it does on Ext4 on Linux, so long times on HDD are unfortunately expected. But you could enable --cache to make subsequent scans much faster.

Mae72DE commented 2 years ago

I only just "C:\" for now.

 ☑ Lokale HDD
 ☒ Network not testing yet - sorry -
    I got some Ext4 in my NAS.
    So a OMV-Plugin of fclone will be nice,
    I think the 'SMB:\\NAS\…' should not work to feel the optimization of Ext4 from my old Windows7-PC.

  FS: NTFS
 RPM: 5400 RPM @ 43°C 

  897.648.549.120 Bytes  used
  102.148.632.576 Bytes  free

 More about "C:\"     
    ST1000LX001-1EM164 
    1000,2 GB  Serial ATA
    SATA/600
    ACS-2 | ACS-3 Revision 3b
    S.M.A.R.T., APM, NCQ 

I'll try the cache today - and hope so to understand where it will store that. And then, I'll try to link the doublettes with "fclone-link"

To use next time the "--unique" Flag, alias "--rf-under 2" I still miss the "fclones.exe-COPY" Sub-Programm, or better just an "action" or "select" Program with an Option like "--transform" from the "group"-Sub - to do what I want with the selected files. But for that option there will come the CodePage-Problem: Win7 Unicode-Support is very bugy - I think Win 10 may also use no real UTF-8 …