MD5 file hashes causing endless download loops in NC 12.0.3

elevatingcreativity commented 6 years ago

### Steps to reproduce 1. Some unknown prior Steps leading up to the problem... 2. Attempt to sync previously existing files in nextcloud to new client machine on MacOS High Sierra or MacOS Sierra with 2.3.2 (build 1) 3. Notice that it loops forever because it cannot properly verify the checksum ### Expected behaviour It should download/sync the files to the new client ### Actual behaviour A large number of files get the error " Nextcloud,The downloaded file does not match the checksum, it will be resumed." For each file that gets this error, it will re-try to download, again and again and again and again and never succeed. Looking more deeply, I checked the database because I initially suspected corruption of the oc_filechache.checksum field. However, it is correct. Here is an example: * In the database checksums of one problem file: ``` SHA1:86e6facad5418fdb693bc0fc4b8e9fac2161f007 MD5:0b93cff7f915a8e52648b1bb52b57a34 ADLER32:777c40bc ``` * On the server checksums of the same file: ``` sudo -u www-data sha1sum 86e6facad5418fdb693bc0fc4b8e9fac2161f007 sudo -u www-data md5sum 0b93cff7f915a8e52648b1bb52b57a34 ``` * On the mac client, after downloading by removing the file hashes from the corresponding cell in the database ``` sha1sum 86e6facad5418fdb693bc0fc4b8e9fac2161f007 md5sum 0b93cff7f915a8e52648b1bb52b57a34 ``` Okay, so scratch head.... these checksums match!? So why is the client saying they don't match? I have no idea. However, one more clue. Try removing checksum data from the particular column/cell for this file in the database: 1. Remove all checksum data: file syncs properly 2. Remove only alder32 data: file does not sync 3. Remove alder32 data and MD5 data, but leave SHA1 data: file syncs Therefore, it appears that the problem is with the MD5 data, for some reason on certain files, it cannot verify. Furthermore, I checked the server logs: sudo -u www-data grep /mnt/ncdata/nextcloud.log There was no entry for the affected files in the server logs. So this appears to be a client issue. As a final step, I tested what happens when I remove an affected file and re-upload it. After doing so, the file downloads fine on all clients. Checking the new database entry, _there is nothing in the checksum field_. Interesting! So I am wondering now what is going on with the checksums, why some files have them and some don't? I was migrated from an older Owncloud install a few months ago, could that be the issue? Even if it is due to the owncloud migration, this should be handled automatically.... ### Server configuration **Operating system**: Ubuntu 16.04.3 LTS **Web server:** Apache/2.4.18 (Ubuntu) **Database:** MariaDB 10.2.9 **PHP version:** 7.0.22-0ubuntu0.16.04.1 **Nextcloud version:** (see Nextcloud admin page) 12.0.3 **Updated from an older Nextcloud/ownCloud or fresh install:** Updated from 12.0.1 (but fresh install before that) **Where did you install Nextcloud from:** Tech and Me VM **Signing status:**

Signing status

``` No errors have been found. ```

**List of activated apps:**

App list

``` Enabled: - activity: 2.5.2 - bruteforcesettings: 1.0.2 - calendar: 1.5.5 - comments: 1.2.0 - contacts: 2.0.1 - dav: 1.3.0 - drawio: 0.8.8 - federatedfilesharing: 1.2.0 - files: 1.7.2 - files_downloadactivity: 1.1.1 - files_external: 1.3.0 - files_markdown: 2.0.1 - files_pdfviewer: 1.1.1 - files_sharing: 1.4.0 - files_texteditor: 2.4.1 - files_trashbin: 1.2.0 - files_versions: 1.5.0 - files_videoplayer: 1.1.0 - firstrunwizard: 2.1 - gallery: 17.0.0 - logreader: 2.0.0 - lookup_server_connector: 1.0.0 - nextant: 1.0.8 - nextcloud_announcements: 1.1 - notifications: 2.0.0 - oauth2: 1.0.5 - password_policy: 1.2.2 - provisioning_api: 1.2.0 - richdocuments: 1.12.35 - serverinfo: 1.2.0 - sharebymail: 1.2.0 - spreed: 2.0.1 - survey_client: 1.0.0 - systemtags: 1.2.0 - theming: 1.3.0 - twofactor_backupcodes: 1.1.1 - updatenotification: 1.2.0 - workflowengine: 1.2.0 Disabled: - admin_audit - encryption - federation - previewgenerator - ransomware_protection - user_external - user_ldap ```

**Nextcloud configuration:**

Config report

``` { "system": { "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "localhost", "***REMOVED SENSITIVE VALUE***", "***REMOVED SENSITIVE VALUE***", "***REMOVED SENSITIVE VALUE***" ], "datadirectory": "\/mnt\/ncdata", "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***\/", "dbtype": "mysql", "version": "12.0.3.3", "dbname": "nextcloud_db", "dbhost": "localhost", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "instanceid": "oc3h4gsr8sgn", "maintenance": true, "mail_smtpmode": "smtp", "memcache.local": "\\OC\\Memcache\\Redis", "filelocking.enabled": true, "memcache.distributed": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "\/var\/run\/redis\/redis.sock", "port": 0, "timeout": 0, "dbindex": 0, "password": "***REMOVED SENSITIVE VALUE***" }, "htaccess.RewriteBase": "\/", "loglevel": "2", "log_type": "file", "logfile": "\/mnt\/ncdata\/nextcloud.log", "logtimezone": "America\/Denver", "mail_from_address": "admin", "mail_smtpauthtype": "LOGIN", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtpauth": 1, "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "theme": "" } } ```

**Are you using external storage, if yes which one:** no **Are you using encryption:** no **Are you using an external user-backend, if yes which one:** I don't fully understand the question... yes, there is webdav access for a backup process, but it has not touched the affected files here. ### Client configuration **Browser:** I've tried Safari and Firefox **Operating system:** MacOS High Sierra **Client** Nextcloud client 2.3.2 (build 1) ### Logs #### Web server error log

Web server error log

``` [Fri Oct 13 06:25:08.906061 2017] [mpm_prefork:notice] [pid 1549] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g configured -- resuming normal operations [Fri Oct 13 06:25:08.906082 2017] [core:notice] [pid 1549] AH00094: Command line: '/usr/sbin/apache2' [Fri Oct 13 08:02:12.395926 2017] [authz_core:error] [pid 45623] [client 184.155.63.23:49752] AH01630: client denied by server configuration: /usr/share/phpmyadmin [Fri Oct 13 08:10:17.113581 2017] [mpm_prefork:notice] [pid 1549] AH00169: caught SIGTERM, shutting down [Fri Oct 13 08:10:18.246375 2017] [mpm_prefork:notice] [pid 47150] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g configured -- resuming normal operations [Fri Oct 13 08:10:18.246435 2017] [core:notice] [pid 47150] AH00094: Command line: '/usr/sbin/apache2'

``` #### Nextcloud log (data/nextcloud.log)

Nextcloud log

I have not included the whole log because it is large. However, after taking the system out of maintenance mode, I ran a new sync attempt on the client, and no logs were produced on the server, even though the client continued to report the same error on these files. ``` {"reqId":"mgPmh0KmxjRdvkKlKy7f","level":4,"time":"2017-10-13T09:17:14-06:00","remoteAddr":"redacted","user":"--","app":"webdav","method":"GET","url":"redacted","message":"Exception: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"System in maintenance mode.\",\"Code\":0,\"Trace\":\"#0 [internal function]: OCA\\\\DAV\\\\Connector\\\\Sabre\\\\MaintenancePlugin->checkMaintenanceMode(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#1 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#2 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(466): Sabre\\\\Event\\\\EventEmitter->emit('beforeMethod', Array)\\n#3 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#4 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(76): Sabre\\\\DAV\\\\Server->exec()\\n#5 \\\/var\\\/www\\\/nextcloud\\\/remote.php(162): require_once('\\\/var\\\/www\\\/nextcl...')\\n#6 {main}\",\"File\":\"\\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/MaintenancePlugin.php\",\"Line\":82}","userAgent":"DEVONcloudy 1.6.5","version":"12.0.3.3"} {"reqId":"amF7oUtWKZtneOhMgCDW","level":4,"time":"2017-10-13T09:17:15-06:00","remoteAddr":"redacted","user":"--","app":"webdav","method":"GET","url":"redacted","message":"Exception: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"System in maintenance mode.\",\"Code\":0,\"Trace\":\"#0 [internal function]: OCA\\\\DAV\\\\Connector\\\\Sabre\\\\MaintenancePlugin->checkMaintenanceMode(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#1 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#2 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(466): Sabre\\\\Event\\\\EventEmitter->emit('beforeMethod', Array)\\n#3 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#4 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(76): Sabre\\\\DAV\\\\Server->exec()\\n#5 \\\/var\\\/www\\\/nextcloud\\\/remote.php(162): require_once('\\\/var\\\/www\\\/nextcl...')\\n#6 {main}\",\"File\":\"\\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/MaintenancePlugin.php\",\"Line\":82}","userAgent":"DEVONcloudy 1.6.5","version":"12.0.3.3"} ```

#### Browser log Not applicable #### Client log This is just a short log extract from the client. ``` 10/13/17 9:53:14 AM, sysadmin/vmware/VMware-Remote-Console.dmg, Nextcloud,The downloaded file does not match the checksum, it will be resumed. ```

ckamm commented 6 years ago

My guess here is that this is a nextCloud server bug. It probably sends multiple checksums when the OC-Checksum header should contain only one (preferably the SHA1).

elevatingcreativity commented 6 years ago

I'm not yet familiar enough with the procedures here @ckamm to know what I should do if you're right - would I need to refile this under the server threads somewhere?

khumarahn commented 6 years ago

Hey, I have the same problem, I think. I switched from owncloud, and it went smooth, there were no problems on server or on clients. Some time later I setup a fresh client, and the download goes in the endless loop.

It there a working solution, even if a temporary one?

drkmccy commented 6 years ago

I'm getting the same on my 12.0.3 box. Tried both the latest Nextcloud client and also the ownCloud client. Both behave the same.

I think this issue should be moved to server, although the mobile apps seem to sync fine.

khumarahn commented 6 years ago

I used a workaround from there: https://help.nextcloud.com/t/windows-sync-client-doesnt-sync-all-files-checksum-wrong/14044/8

jospoortvliet commented 6 years ago

The Woboq team was so nice to help us fix this in https://github.com/owncloud/client/pull/6172 - we'll get it in our client soon.

ulistaerk commented 6 years ago

I had this issue, too.

I have a different statment to remove the mis-behaving checksums: update oc_filecache set checksum=substring(checksum, 1, 45) from oc_filecache where checksum LIKE 'SHA1:%';

Morodar commented 6 years ago

I know this repo is related to nextcloud, but ...

Same problem happened to me using ownCloud Sync Client (and ownCloud Server).

Fix that worked for me

Typing (in MariaDB) while owncloud_db is used.

UPDATE oc_filecache
SET checksum = ''
WHERE COALESCE (checksum, ‘’) <> ‘’;

Fixed the problem for me. I found the fix here as reported by khumarahn

Server Configuration

Operating system: Ubuntu 16.04.4 LTS

Web server: nginx /1.10.3 (Ubuntu)

Database: MariaDB 10.0.34

PHP version: v7.0.28-0ubuntu0.16.04.1

ownCloud version: ownCloud - 10.0.8 (stable)

ownCloud Client: ownCloud-client 2.4.1.9270

When it happened

All files which I have deleted and then restored showed the error The downloaded file does not match the checksum, it will be resumed

Maybe this Information helps.

ckamm commented 6 years ago

@Morodar This seems like a different issue, unrelated to the server sending multiple content hashes at once. Can you explain about how you created the situation in which you got the "does not match the checksum" error in detail? How exactly did you delete and restore files?

Preferably create a new bug report at https://github.com/owncloud/client/issues .

Morodar commented 6 years ago

@ckamm Apparently, I wasn't able to reproduce the error. I tried a couple of deletions and restorations on multiple devices but everything worked correctly.

The only thing I can remember is that I've reset my Android phone. After reset, I downloaded the App FolderSync and configured to sync a couple of Folders (Documents, Music, Pictures, WhatsApp). These Folders are also synchronized with my Workstation.

(FolderSync is able to Sync Directories between your phone and GoogleDrive, Amazon, ... , and also ownCloud)

Somehow FolderSync deleted about 400 .mp3 files. These 400 .mp3 have been deleted on my Workstation too.

I restored them in the Browser in the ownCloud 'Deleted Files' section (CTRL-Shift is not working, so it took a while).

After that, my Workstation downloaded all restored .mp3 files. I was wondering why I hear my drives more often then before. (Server is next to me)

I opened ownCloud Sync Client which showed me The downloaded file does not match the checksum, it will be resumed error for these 400 .mp3 files. ... Then I realized that I had the Download loop.

That's the reason why i though that the File deletion and restoration may have caused the error.

nextcloud / client_theming