[Windows] client doesn’t support with Windows deduplication (sym links)

falkenau commented 8 years ago

Hi there!

Expected behaviour

Owncloud Windows client should sync all files.

Actual behaviour

Not all files are synced.

Steps to reproduce

You need a Windows 8.1 with an extra partition.
On the extra partition (usually d:\ ) format with NTFS and configure deduplication. (https://weikingteh.wordpress.com/2013/10/10/enable-deduplication-on-your-windows-8-1/
Make sure that deduplication had done good job. (see the same hyperlink)
Install Owncloud client and let it sync lot of files form volume d:\
Server configuration

Operating system: Linux (QNAP NAS)

Web server: Unknown

Database: Unknown

PHP version: Unknown

ownCloud version: 8.0.4

Storage backend: Unknown

Client configuration

Client version: Owncloud client Version 2.0.2 (build 5569)

Operating system: Windows 8.1 Pro x64

OS language: German

Installation path of client: C:\

Logs

It turns out that there are many errors of the same kind: (0x3778a8) 11-02 01:48:41:776 0x3778a8 OCC::SyncEngine::slotItemCompleted: void OCC::SyncEngine::slotItemCompleted(const OCC::SyncFileItem&, const OCC::PropagatorJob&) "Scan-Laptop.pdf" INSTRUCTION_IGNORE 6 "Symbolische Verknüpfungen werden bei der Synchronisation nicht unterstützt."

(Symbolic links are not supported in syncing)

Full story:

Windows Owncloud client doesn’t support with Windows deduplication

I’m using Windows 8.1 Pro. On my second volume I’m using NTFS with deduplication. (https://weikingteh.wordpress.com/2013/10/10/enable-deduplication-on-your-windows-8-1/)

Now I bought a QNAP NAS and installed Owncloud 8.0.4. Almost everything works very well, Android client on my smartphone, LDAP connection and a test Windows client.

However, my private notebook with the deduplicated volume, doesn’t work. A lot of files are synced to Owncloud and almost half of the files are not synced.

I used the logwindow option to get more information: owncloud.exe --logwindow --logfile c:\temp\OCclient.log

It turns out that there are many errors of the same kind: (0x3778a8) 11-02 01:48:41:776 0x3778a8 OCC::SyncEngine::slotItemCompleted: void OCC::SyncEngine::slotItemCompleted(const OCC::SyncFileItem&, const OCC::PropagatorJob&) "Scan-Laptop.pdf" INSTRUCTION_IGNORE 6 "Symbolische Verknüpfungen werden bei der Synchronisation nicht unterstützt."

(Symbolic links are not supported in syncing)

The file named “Scan-Laptop.pdf” has a blue mark, not a green one. All errors are tagged 0x3778a8. If I copy the file it will get synced. But after Windows deduplication has deduped the copy, it will get a blue mark and deleted from Owncloud.

Junction Link Magic (http://www.rekenwonder.com/linkmagic.htm) is not able to find any symlink on the second volume D:. Of course there are many in C:\ (my systemroot).

Therefore, I guess, the Owncloud client is seeing all my deduplicated files as symbolic links. And so the client is pretty useless to me. :(

Any suggestions? Should this be treated as a bug or as a feature request?

Hope it helps.

phil-davis commented 8 years ago

@falkenau Do you know what type of NTFS links the deduplication package makes in the file system to point 2 folder/file entries to the same physical storage? (short cut (hopefully not), symbolic link, junction point - and if 2 directories are identical does it make directory symbolic link or directory junction...) That will help in knowing the necessary link types that would need to be followed to support this. For reference, the various options to the "mklink" command in Windows: C:\Users\phil.davis>mklink Creates a symbolic link.

MKLINK [[/D] | [/H] | [/J]] Link Target

    /D      Creates a directory symbolic link.  Default is a file
            symbolic link.
    /H      Creates a hard link instead of a symbolic link.
    /J      Creates a Directory Junction.
    Link    Specifies the new symbolic link name.
    Target  Specifies the path (relative or absolute) that the new link
            refers to.

falkenau commented 8 years ago

I know mklink. But mklink doesn’t list all existing links.

Junction Link Magic (http://www.rekenwonder.com/linkmagic.htm) is a good tool to get a list of all different types of links on your NTFS volume. But as I wrote, there are none.

I think it would not be possible to handle deduplication with symbolic link or hard links, because Windows is using a “block level deduplication” https://technet.microsoft.com/en-us/library/hh831602.aspx

“small variable-sized chunks (32–128 KB), identifying duplicate chunks, and maintaining a single copy of each chunk”

mklink could not provide this feature.

With “perfmon /res” I can see, that deduplicated files read from and wrote to “\System Volume Information\Dedup\ChunkStore*”.

phil-davis commented 8 years ago

@falkenau Yes, you are correct. It is doing lower-level dedup than the file system. From the docs I see that it is doing something different in the file header meta-data to point to the various data chunks in the chunk store. For applications that use the standard IO calls to read and write the file data, it should be transparent. I wonder what the ownCloud Client is doing special, what file attribute/metadata setting it is checking that makes it not so transparent?

falkenau commented 8 years ago

@phil-davis Well, I tried to find a tool to show me the metadata of NTFS files.

Finding a good freeware tool is a hard task. Finally, I found AccessData FTK Imager: http://accessdata.com/product-download/digital-forensics/ftk-imager-version-3.4.2

And I found a pattern.

All files that are synced with Owncloud client have two attributes more “Start Cluster” and “Start Sector”. All files that are not synced seems to miss this attributes.

In addition, the synced file are all labeled as: "File Class":"Regular File" Not synced files are labeled: “File Class”:”Reparse Point" https://en.wikipedia.org/wiki/NTFS_reparse_point

If anybody could suggest a better tool, I would take a deeper look.

phil-davis commented 8 years ago

I expect that the "Regular file" ones are the files that have had no opportunity for dedup. And so they are still regular files in the file system and thus ownCloud sees them and syncs them. "Reparse Point" will be the files that have had some dedup applied. I guess that ownCloud is seeing the "Reparse Point" attributes and because of that thinking that this is a link or "other weird thing" and thus ignoring for sync. I will troll around a bit in the code to educate myself about what it is doing, but if there is someone who actually knows and can go quickly to it then please add your input...

Takuji-S commented 8 years ago

Hi, I have used the ownCloud windows client on Windows 2012 with dedupe volume. It seems to be working fine in ownCloud-2.0.1.5446. However the newer version than 2.0.1.5446 seems to be unable to handle symbolic-link.