nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.5k stars 3.98k forks source link

Auto upload to external storage fails #13297

Closed wernerfred closed 4 years ago

wernerfred commented 5 years ago

Steps to reproduce

  1. Add external storage via webdav
  2. Setup photo auto upload to folder inside this external storage
  3. Try to (auto) upload files with small size ~kB range

Expected behaviour

I expect nextcloud app to upload the pictures to the defined folder at the external storage

Actual behaviour

Upload runs to 100% and then fails with "unknown error" or "server misconfigured"

Server configuration

Operating system: Debian 9

Web server: Apache2

Database: mysql

PHP version: 7.0

Nextcloud version: (see Nextcloud admin page) 15.0.0

Updated from an older Nextcloud/ownCloud or fresh install: fresh install

Where did you install Nextcloud from: official homepage

Signing status:

Signing status ``` No errors have been found. ```

List of activated apps:

App list ``` Enabled: - accessibility: 1.1.0 - activity: 2.8.2 - bruteforcesettings: 1.3.0 - cloud_federation_api: 0.1.0 - comments: 1.5.0 - dav: 1.8.0 - external: 3.2.0 - federatedfilesharing: 1.5.0 - federation: 1.5.0 - files: 1.10.0 - files_external: 1.6.0 - files_pdfviewer: 1.4.0 - files_sharing: 1.7.0 - files_texteditor: 2.7.0 - files_trashbin: 1.5.0 - files_versions: 1.8.0 - files_videoplayer: 1.4.0 - firstrunwizard: 2.4.0 - gallery: 18.2.0 - logreader: 2.0.0 - lookup_server_connector: 1.3.0 - nextcloud_announcements: 1.4.0 - notifications: 2.3.0 - oauth2: 1.3.0 - password_policy: 1.5.0 - provisioning_api: 1.5.0 - serverinfo: 1.5.0 - sharebymail: 1.5.0 - support: 1.0.0 - survey_client: 1.3.0 - systemtags: 1.5.0 - theming: 1.6.0 - twofactor_backupcodes: 1.4.1 - updatenotification: 1.5.0 - workflowengine: 1.5.0 Disabled: - admin_audit - encryption - user_external - user_ldap ```

Nextcloud configuration:

Config report ``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "my.domain", ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "15.0.0.10", "overwrite.cli.url": "https:\/\/my.domain", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "memcache.local": "\\OC\\Memcache\\APCu", "mail_smtpmode": "sendmail", "mail_smtpsecure": "tls", "mail_sendmailmode": "smtp", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "maintenance": false } } ```

Are you using external storage, if yes which one: webdav

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Chrome 70

Operating system: Debian 9

Logs

Web server error log

Web server error log ``` IP - wernerf [30/Dec/2018:20:08:39 +0100] "GET /index.php/204 HTTP/1.1" 204 4449 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1" IP - wernerf [30/Dec/2018:20:08:50 +0100] "PROPFIND /remote.php/dav/files/wernerf/ HTTP/1.1" 207 1376 "-" "Mozilla/5.0 (Linux) mirall/2.5.1git (Nextcloud)" IP - - [30/Dec/2018:20:08:53 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 984 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36" IP - wernerf [30/Dec/2018:20:08:40 +0100] "PUT /remote.php/webdav/sigma/photo/Whatsapp//2018/12/IMG-20181229-WA0000.jpg HTTP/1.1" 500 888 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1" IP - wernerf [30/Dec/2018:20:09:03 +0100] "PROPFIND /remote.php/dav/files/wernerf/ HTTP/1.1" 207 1377 "-" "Mozilla/5.0 (Linux) mirall/2.5.1git (Nextcloud)" ```

Nextcloud log (data/nextcloud.log)

Nextcloud log ``` Fatal | webdav | Sabre\DAV\Exception: Could not rename part file to final file/srv/http/cloud.my.domain/htdocs/apps/dav/lib/Connector/Sabre/Directory.php - line 156:OCA\DAV\Connector\Sabre\File->put(null)/srv/http/cloud.my.domain/htdocs/3rdparty/sabre/dav/lib/DAV/Server.php - line 1096:OCA\DAV\Connector\Sabre\Directory->createFile("IMG-20181230-WA0006.jpg", null)/srv/http/cloud.my.domain/htdocs/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 525:Sabre\DAV\Server->createFile("sigma/photo ... g", null, null)<>Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\R ... "}, Sabre\HTTP\Response {})/srv/http/cloud.my.domain/htdocs/3rdparty/sabre/event/lib/EventEmitterTrait.php - line 105:call_user_func_array([ Sabre\DAV\ ... "], [ Sabre\HTTP ... }])/srv/http/cloud.my.domain/htdocs/3rdparty/sabre/dav/lib/DAV/Server.php - line 479:Sabre\Event\EventEmitter->emit("method:PUT", [ Sabre\HTTP ... }])/srv/http/cloud.my.domain/htdocs/3rdparty/sabre/dav/lib/DAV/Server.php - line 254:Sabre\DAV\Server->invokeMethod(Sabre\HTTP\R ... "}, Sabre\HTTP\Response {})/srv/http/cloud.my.domain/htdocs/apps/dav/appinfo/v1/webdav.php - line 80:Sabre\DAV\Server->exec()/srv/http/cloud.my.domain/htdocs/remote.php - line 163:require_once("/srv/http/c ... p") |   | 2018-12-30T20:04:41+0100 -- | -- | -- | -- | -- Error | webdav | renaming part file to final file failed $renameOkay: false, $fileExists: false) ```
wernerfred commented 5 years ago

I figured out that auto uploading of bigger files works flawless. If i took a picture with my smartphone camera (~5-15MB) it will upload the file in multiple steps and then save the file at its destination location. But with pictures received from whatsapp (~50-800kB) it will result in an error with the message above:

Sabre\DAV\Exception: Could not rename part file to final file

It seems like Nextcloud thinks that the picture exists of multiple chunks which should be then merged together again but because the whatsapp picture is so small it consists only of one "chunk" and by trying to merge it the secound "chunk" is null and therefore it fails.

Could this be the cause? Any further information/logs needed?

Also find out that if a upload is successful the webserver log looks like this:

IP - wernerf [31/Dec/2018:13:04:09 +0100] "PUT /remote.php/webdav/Uploads//2018/12/IMG-20181231-WA0007.jpg HTTP/1.1" 201 766 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:04:10 +0100] "PROPFIND /remote.php/webdav/Uploads//2018/12/IMG-20181231-WA0007.jpg HTTP/1.1" 207 1560 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"

And if it fails, it looks like this:

IP - wernerf [31/Dec/2018:13:04:01 +0100] "PUT /remote.php/webdav/sigma/photo/Whatsapp//2018/12/IMG-20181230-WA0013.jpg HTTP/1.1" 500 888 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:04:03 +0100] "GET /index.php/204 HTTP/1.1" 204 1521 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:04:04 +0100] "PROPFIND /remote.php/dav/files/wernerf/ HTTP/1.1" 207 1376 "-" "Mozilla/5.0 (Linux) mirall/2.5.1git (Nextcloud)"

So the log says in the correct way:

PROPFIND /remote.php/webdav/Uploads//2018/12/IMG-20181231-WA0007.jpg 

and if it fails

PROPFIND /remote.php/dav/files/wernerf/ 

Maybe this will help solving the issue

wernerfred commented 5 years ago

I tried uploading 2 different pictures (102KB and 4MB) manually to the same external storage location and watched for the logs.

102KB upload which results in error:

IP - wernerf [31/Dec/2018:13:44:39 +0100] "GET /index.php/204 HTTP/1.1" 204 1519 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:44:39 +0100] "HEAD /remote.php/webdav/sigma/photo/Whatsapp/2018/12/IMG-20181231-WA0009.jpg HTTP/1.1" 404 613 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - - [31/Dec/2018:13:44:40 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 984 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36"
IP - wernerf [31/Dec/2018:13:44:40 +0100] "PUT /remote.php/webdav/sigma/photo/Whatsapp/2018/12/IMG-20181231-WA0009.jpg HTTP/1.1" 500 888 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:44:59 +0100] "PROPFIND /remote.php/dav/files/wernerf/ HTTP/1.1" 207 1377 "-" "Mozilla/5.0 (Linux) mirall/2.5.1git (Nextcloud)"

4MB upload which runs without error:

IP - wernerf [31/Dec/2018:13:46:23 +0100] "GET /index.php/204 HTTP/1.1" 204 1382 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:46:23 +0100] "HEAD /remote.php/webdav/sigma/photo/Whatsapp/2018/12/IMG_20181230_141328.jpg HTTP/1.1" 404 613 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:46:23 +0100] "PROPFIND /remote.php/webdav/sigma/photo/Whatsapp/2018/12/ HTTP/1.1" 207 1519 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:46:24 +0100] "PROPFIND /remote.php/webdav/sigma/photo/Whatsapp/2018/12/ HTTP/1.1" 207 356555 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:46:23 +0100] "PUT /remote.php/webdav/sigma/photo/Whatsapp/2018/12/IMG_20181230_141328.jpg-chunking-199813246-4-0 HTTP/1.1" 201 624 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:46:25 +0100] "PUT /remote.php/webdav/sigma/photo/Whatsapp/2018/12/IMG_20181230_141328.jpg-chunking-199813246-4-1 HTTP/1.1" 201 624 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:46:27 +0100] "PUT /remote.php/webdav/sigma/photo/Whatsapp/2018/12/IMG_20181230_141328.jpg-chunking-199813246-4-2 HTTP/1.1" 201 624 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:46:28 +0100] "GET /ocs/v2.php/apps/files_sharing/api/v1/shares?path=%2Fsigma%2Fphoto%2FWhatsapp%2F2018%2F12%2F&reshares=true&subfiles=true HTTP/1.1" 200 823 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:46:28 +0100] "PUT /remote.php/webdav/sigma/photo/Whatsapp/2018/12/IMG_20181230_141328.jpg-chunking-199813246-4-3 HTTP/1.1" 500 896 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:47:03 +0100] "GET /index.php/204 HTTP/1.1" 204 1380 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:47:03 +0100] "HEAD /remote.php/webdav/sigma/photo/Whatsapp/2018/12/IMG_20181230_141328.jpg HTTP/1.1" 404 613 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:47:04 +0100] "PROPFIND /remote.php/dav/files/wernerf/ HTTP/1.1" 207 1376 "-" "Mozilla/5.0 (Linux) mirall/2.5.1git (Nextcloud)"
IP - wernerf [31/Dec/2018:13:47:04 +0100] "PUT /remote.php/webdav/sigma/photo/Whatsapp/2018/12/IMG_20181230_141328.jpg-chunking-199813246-4-3 HTTP/1.1" 201 624 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:47:05 +0100] "PROPFIND /remote.php/webdav/sigma/photo/Whatsapp/2018/12/IMG_20181230_141328.jpg HTTP/1.1" 404 961 "-" "Mozilla/5.0 (Android) ownCloud-android/3.4.1"
IP - wernerf [31/Dec/2018:13:47:07 +0100] "PROPFIND /remote.php/dav/files/wernerf/ HTTP/1.1" 207 1234 "-" "Mozilla/5.0 (Linux) mirall/2.5.1git (Nextcloud)"

I think that the error is not caused by the webdav storage as the 4MB file is uploaded to the exact same location. The difference between the two uploads are only the file sizes, i even testet uploading the same image, one raw and one compressed.

Any idea?

wernerfred commented 5 years ago

Same Issue with same Error messages on Linux Desktop Client, seems to be a general problem with chunking of small files.

erdoukki commented 5 years ago

I have the same problem in an ARM installation... iOS APP looping at uploading photos to my NextCloud instance.

I am using a NextCloudPi scripted installation.

My external storage is a nfs remote share, from a NetGear NAS which was mounted with the nolock option. I was badly thinking this option was making the looping of small files at synch/upload.

But if I mount without the nolock options, the mass upload of my iPhone look working better but finally end in a loop of small files getting the queue full.

IP - - [05/Feb/2019:18:41:06 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2012-38-28%201389.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:06 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2019-19-31%201428.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:06 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2013-15-34%201393.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:06 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2019-39-39%201434.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:22 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2012-38-28%201389.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:22 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2013-15-34%201393.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:22 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2019-19-31%201428.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:22 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2019-39-39%201434.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:29 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/2.0" 200 547 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0"
IP - - [05/Feb/2019:18:41:35 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2012-38-28%201389.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:35 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2013-15-34%201393.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:35 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2019-19-31%201428.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:35 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2019-39-39%201434.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:49 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2012-38-28%201389.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:49 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2013-15-34%201393.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:49 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2019-19-31%201428.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:50 +0100] "PUT /remote.php/webdav/PHOTOS/IMAGES/ZIFON/2018/11/18-11-21%2019-39-39%201434.jpg HTTP/2.0" 204 0 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.8"
IP - - [05/Feb/2019:18:41:59 +0100] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/2.0" 200 547 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0"

Hope this may help to fix this issues, which is also stated in the iOS APP issue...

skjnldsv commented 4 years ago

As there is no feedback since a while I will close this ticket. If this is still happening please make sure to upgrade to the latest version. After that, feel free to reopen.

juhi24 commented 1 month ago

I'm experiencing a similar issue. Autoupload from andoid app to external webdav-storage fails. It claims that the server is in maintenance mode, but it's definitely not. Upload to internal storage works flawlesly.