search

owncloud / client

🖥️ Desktop Syncing Client for ownCloud
GNU General Public License v2.0
1.4k stars 665 forks source link

Moving files with (Windows-)Client can break Synchronization and may result in dataloss #4138

Closed xn--nding-jua closed 8 years ago

xn--nding-jua commented 8 years ago

Steps to reproduce

  1. Create a top-folder and make it a public sharing within ownCloud webinterface
  2. Create a new folder within this public folder (Windows names it "Neuer Ordner")
  3. Give the folder a desired name (e.g. folder1)
  4. Copy some files into folder1 (in my case around 20 mp3s with each ~5 MB)
  5. Create a second new folder and give it a new name (e.g. folder2)
  6. Use Ctrl+X -> Ctrl+V to move the mp3s from folder1 to folder2
  7. Create a third and fourth folder and rename it (e.g. folder3 and - guess it - folder4)
  8. Move some files from folder2 to folder3 and some files to folder4

    Expected behaviour

When synchronization has finished all files should be stored within ownCloud like in the Windows-directory-structure.

Actual behaviour

The Windows-Client stops working with several errors (not at the same time):

Some description and hints for the analysis: Error 1: During moving the files the connection interrupted (don't know the reason - internet connection is stable and fast). From one single file only the file within USER/files/folder1 had been moved to USER/files/folder2. The entry in oc_filecache was still USER/files/folder1. The encryption keys were not moved and still in USER/files_encryption/keys/files/folder1. So the client could not found the (encrypted) file anymore and stopped working. I had to move the file per ftp back to the original folder -> client worked again and everything fine for that moment.

Error 2: then the client stopped again. Looking in the ftp-folderstructure the files seemed to be intact, but in oc_filecache I had a double-entry for a file: one entry for the source-folder USER/files/folder1/file.abc and one entry for the destination-folder USER/files/folder2/file.abc (if I remember correctly the etags were the same). After deleting the defective second entry the client started synchronizing again.

Error 3: at the end the client stopped again for the last 5 files. Here it was very strange: all 5 files were missing in the ftp-server but the encryption keys were there and the database-entries in oc_filecache were present. I had to delete the oc_filecache entries and the encryption-folders. Afterwards I had to move the files out of the windows-folder-structure to let the client know, that I want to delete them. After finishing synchronization I could move them back to the cloud-folder and the synchronization-process worked fine again.

If I had no access to oc_filecache/FTP and could not delete/move the defective entries/files the synchronization would not work anymore.

Summa summarum: moving files seems to be problematic at the moment (for multiple files around 5 MB each file). Maybe it is just a timing-problem: I've renamed some folders quite fast, but this should not lead to the three errors described above.

Server configuration

Operating system: Ubuntu 14.04.2 LTS

Web server: Apache (unknown version)

Database: SQL 5.4

PHP version: PHP 5.6

ownCloud version: (see ownCloud admin page) 8.1.0.8

Updated from an older ownCloud or fresh install: from an older installation (8.0.2)

List of activated apps:

The content of config/config.php:

<?php $CONFIG = array ( 'instanceid' => 'xxxx', 'passwordsalt' => 'xxxx', 'trusteddomains' => array ( 0 => 'www.xxxx.de', 1 => 'cloud.xxxx.de', 2 => 'xxxx.de', ), 'datadirectory' => '/www/htdocs/xxxx/xxxx/cloud/data', 'dbtype' => 'mysql', 'version' => '8.1.0.8', 'dbname' => 'xxxx', 'dbhost' => 'localhost', 'dbtableprefix' => 'oc', 'dbuser' => 'xxxx', 'dbpassword' => 'xxxx', 'installed' => true, 'theme' => '', 'maintenance' => false, 'mail_from_address' => 'xxxx', 'mail_smtpmode' => 'php', 'mail_domain' => 'xxxx.de', 'forcessl' => true, 'trashbin_retention_obligation' => 60, 'trashbin_auto_expire' => true, 'loglevel' => 2, 'log_rotate_size' => false, 'logtimezone' => 'Europe/Berlin', 'check_for_working_webdav' => true, 'check_for_working_htaccess' => true, 'xframe_restriction' => false, 'secret' => 'xxxx', 'forceSSLforSubdomains' => true, 'tempdirectory' => '/www/htdocs/xxxx/tmp', 'enable_previews' => true, 'enabledPreviewProviders' => array ( 0 => 'OC\Preview\Image', 1 => 'OC\Preview\MP3', 2 => 'OC\Preview\TXT', 3 => 'OC\Preview\MarkDown', 4 => 'OC\Preview\Illustrator', 5 => 'OC\Preview\Postscript', 6 => 'OC\Preview\Photoshop', 7 => 'OC\Preview\TIFF', ), 'singleuser' => false, );

Are you using external storage, if yes which one: local/smb/sftp/... no

Are you using encryption: yes/no yes

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/... no

Client configuration

ownCloud Client 2.0.2 Browser: Firefox 42.0

Operating system: Windows 7 64-Bit

xn--nding-jua commented 8 years ago

New Error occured:

Error 4: the database-entries were inserted, but no files were on the server. So for the client on the uploading computer everything was fine. But on a second computer the client collapsed, because no files were there for download. What a mess.

Is this chaos on my setup only? Is there something changed on the files-moving-code in ownCloud 8.2?

thanks for any support - from now on I will not move a single file if I cannot be shure that the server can handle this operation correctly...

moscicki commented 8 years ago

Looks like your server is broken. You may want to link this in server bugtracker or ask in the support forums. The client should reasonable follow the moves except for one well-defined case AFAIK (I am not owncloud developer).

PVince81 commented 8 years ago

Are you using php-fpm ? I heard that php-fpm kills the PHP process whenever the connection gets closed (for whatever reason it got closed). If yes, this might explain why the move operation didn't finish on the server side. Another possible explanation is a timeout, but that one is less likely since the sync client would move the files one by one instead of with a single request.

Another possibility is about concurrency: several clients accessing the same instance at the same time while a move is in progress. To prevent potential concurrency issues, a new feature called "transactional file locking" has been introduced: https://doc.owncloud.org/server/8.1/admin_manual/configuration_files/files_locking_transactional.html

Since 8.2 this is enabled by default. (it was still experimental in 8.1 and only had redis support, now in 8.2 it can also store the lock information in the database)

Another idea is to set "filesystem_check_changes" to 0 in config.php to prevent concurrent processes to try and scan the data folder. When encryption is enabled, it doesn't make much sense to leave this enabled anyway. For this reason, this setting is set to 0 by default starting with 8.2.

xn--nding-jua commented 8 years ago

@PVince81 Yes, I'm using FPM/FastCGI because of the access-rights on my shared-hoster. In Apache-Mode I cannot access my data-directory for the monthly backups. I get "Connection has been closed" messages some times. But I had no problems with his instead the files-moving-problem.

"transactional file locking" will be implemented next weekend. When version 8.2.1 is out I will upgrade my server. Afterwards I will stress the server with moving mp3-test-files and I will see if the problem has been solved.

Switching to Apache-mode would be a problem, because I would have to switch all files to ftp-user when I want to make the backup and switch them back to the www-user afterwards. Would not be a good solution for me as the backup is automated.

"filesystem_check_changes" is set to 0 already as I had problems with encryption<->filesize.

ogoffart commented 8 years ago

It seems like your problem caused by a configuration problem in the server rather than a bug in the client. You can ask for help in the forum or other recources from https://owncloud.org/support/