Open ylangisc opened 6 years ago
After some more testing I've noticed that it's even worse. It does not matter if you create a 0 byte file prior updating it. Basically all PUT
operations with a chunked transfer encoding lead to 0 byte files in low latency and high bandwidth environments for files that are greater than a certain size (see above). In my local environment it's easy to replicate with a simple curl command. E.g.
curl -v -X PUT --header "Transfer-Encoding: chunked" -d @testfile.bin "http://ubuntu.local/remote.php/webdav/testfile.bin" -u admin:admin
Hey, this issue has been closed because the label stale
is set and there were no updates for 14 days. Feel free to reopen this issue if you deem it appropriate.
(This is an automated comment from GitMate.io.)
Please reopen this issue.
I too am having this issue with webdav. After disabling encryption and sniffing the traffic going to and from the nextcloud webserver (nginx 1.14.0 in my case), I too came to the conclusion, that after a PUT command with chunked transfer encoding, nextcloud responses with 204 no content and creates only 0 byte files.
I noticed this behavior when trying to export my Kodi library via webdav to nextcloud.
I can also provide a sanitized HTTP communication if needed, but I guess that will not help much.
I just tested on nextcloud 15 with a 700 kb image. 256 kb of the image made it to the server. a 7.5 mb ODP file was also cut in half - 3.3 mb on the server.
curl -v -X PUT --header "Transfer-Encoding: chunked" -d @test.odp "http://127.0.0.1/nextcloud-15/remote.php/webdav/test.odp" -u jos:jos
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'jos'
> PUT /nextcloud-15/remote.php/webdav/test.odp HTTP/1.1
> Host: 127.0.0.1
> Authorization: Basic am9zOmpvcw==
> User-Agent: curl/7.62.0
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* Signaling end of chunked upload via terminating chunk.
< HTTP/1.1 201 Created
< Date: Thu, 03 Jan 2019 11:22:50 GMT
< Server: Apache
< X-Powered-By: PHP/7.3.0
< Set-Cookie: oczswzpgpvdq=9qbdh7bhvlq6s9um76vqfs7dlg; path=/nextcloud-15; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=I1gqnptmzWjmuBJhZqrfPOeNr6LIhTAn6OWTsFxQhflPr2EuxY2LtSmdoIMc0bz4ehfVuMBRmLO%2FZBBQG6FEDfgc0Q7qupNRWiHZPR%2BxESiNJg5Vh1%2Fajn6geMMRJNWv; path=/nextcloud-15; HttpOnly
< Content-Security-Policy: default-src 'none';
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: no-referrer
< Set-Cookie: nc_sameSiteCookielax=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Set-Cookie: oczswzpgpvdq=pj5kbp17hlf3btfob60d2meh7q; path=/nextcloud-15; HttpOnly
< Set-Cookie: cookie_test=test; expires=Thu, 03-Jan-2019 12:22:50 GMT; Max-Age=3600
< OC-FileId: 00001151oczswzpgpvdq
< Content-Length: 0
< ETag: "7ef58d6f933a4e3387fdf1418f188772"
< OC-ETag: "7ef58d6f933a4e3387fdf1418f188772"
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host 127.0.0.1 left intact
I just tested on nextcloud 15 with a 700 kb image. 256 kb of the image made it to the server. a 7.5 mb ODP file was also cut in half - 3.3 mb on the server.
curl -v -X PUT --header "Transfer-Encoding: chunked" -d @test.odp "http://127.0.0.1/nextcloud-15/remote.php/webdav/test.odp" -u jos:jos * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 80 (#0) * Server auth using Basic with user 'jos' > PUT /nextcloud-15/remote.php/webdav/test.odp HTTP/1.1 > Host: 127.0.0.1 > Authorization: Basic am9zOmpvcw== > User-Agent: curl/7.62.0 > Accept: */* > Transfer-Encoding: chunked > Content-Type: application/x-www-form-urlencoded > Expect: 100-continue > < HTTP/1.1 100 Continue * Signaling end of chunked upload via terminating chunk. < HTTP/1.1 201 Created < Date: Thu, 03 Jan 2019 11:22:50 GMT < Server: Apache < X-Powered-By: PHP/7.3.0 < Set-Cookie: oczswzpgpvdq=9qbdh7bhvlq6s9um76vqfs7dlg; path=/nextcloud-15; HttpOnly < Expires: Thu, 19 Nov 1981 08:52:00 GMT < Cache-Control: no-store, no-cache, must-revalidate < Pragma: no-cache < Set-Cookie: oc_sessionPassphrase=I1gqnptmzWjmuBJhZqrfPOeNr6LIhTAn6OWTsFxQhflPr2EuxY2LtSmdoIMc0bz4ehfVuMBRmLO%2FZBBQG6FEDfgc0Q7qupNRWiHZPR%2BxESiNJg5Vh1%2Fajn6geMMRJNWv; path=/nextcloud-15; HttpOnly < Content-Security-Policy: default-src 'none'; < X-Frame-Options: SAMEORIGIN < X-XSS-Protection: 1; mode=block < X-Content-Type-Options: nosniff < X-Robots-Tag: none < X-Download-Options: noopen < X-Permitted-Cross-Domain-Policies: none < Referrer-Policy: no-referrer < Set-Cookie: nc_sameSiteCookielax=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax < Set-Cookie: nc_sameSiteCookiestrict=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict < Set-Cookie: oczswzpgpvdq=pj5kbp17hlf3btfob60d2meh7q; path=/nextcloud-15; HttpOnly < Set-Cookie: cookie_test=test; expires=Thu, 03-Jan-2019 12:22:50 GMT; Max-Age=3600 < OC-FileId: 00001151oczswzpgpvdq < Content-Length: 0 < ETag: "7ef58d6f933a4e3387fdf1418f188772" < OC-ETag: "7ef58d6f933a4e3387fdf1418f188772" < Content-Type: text/html; charset=UTF-8 < * Connection #0 to host 127.0.0.1 left intact
I have been doing some related tests on another webdav service with curl, and for me it started to work ok when using other Content-Type header, like "application/octet-stream" instead of "application/x-www-form-urlencoded".
Also running into this issue on Nextcloud 16.04. Nginx 1.16.1
macOS finder DAV client.
With "fastcgi_request_buffering off" uploads in Finder to the Nextcloud server over a DAV mount always end up "0 size".
If we enable "fastcgi_request_buffering on" it's hit or miss. Smaller files generally work, but larger files almost always end up truncated. (0 size).
Is this getting worked on? I would appreciate an update apart from the modified milestones. If I can help with debugging in any way (the first few posts do a good job on explaining how to reproduce the issue, IMHO), please let me know. Its quite an awful feeling that a part of my data gets destroyed once I use something like the iOS-App to Update files and it keeps me away from using anything other than the webinterface.
@Subito sorry, not assigned yet.
I cannot push data from my QNAP with Hybrid Backup Sync, due to this, have not found any option to unchunk it yet.
This bug is still present with nginx 1.18.0.
With Apache + PHP-FPM, chunked transfers destroy all of my larger files very reliably. But that seems to be related to a different bug: https://bz.apache.org/bugzilla/show_bug.cgi?id=57087
So I've put nginx behind an Apache reverse proxy and then set these options in the nginx configuration so it doesn't error out on uploading larger files:
client_max_body_size 50000M;
client_body_buffer_size 400M;
fastcgi_buffers 64 4K;
I've tried uploading 30 files that are under <10k and half of them ended up being 0 bytes.
With fastcgi_request_buffering on
only one time a file was truncated to 0 bytes in three subsequent test runs and very large files all transferred fine, too.
So while that option makes it a LOT better it is still not reliable and I cannot trust my data. mod_php being the only option to run Nextcloud reliably makes this very unsuitable for a VPS. It's already a bit painful that I have to run two web servers to make it almost work, but the blame on that can be put on the Apache team too, I guess.
I hope someone will fix this soon.
We have documented our deployment workarounds in Zero byte file truncate issue with Nextcloud and ownCloud deployed with FastCGI.
Adding myself @j-dimension to the list of watchers. Happens to me with Nextcloud 18.06
@dkocher thanks for the tip, I enabled this too on my Nginx Nextcloud config which improved the situation. Originally the documentation for Nextcloud calls for this fastcgi_request_buffering to be turned off. I had to turn it on to improve the situation since I'm waiting for NC19.0.1 to hopefully better address these type of issues. So much tinkering is required to make this setup work.
Nevermind, I spoke too soon, it still is happening. The file uploads properly to the data directory but webdav/webui still report it as 0KB. The only option I have to run a files:scan --all but that takes forever for 50TB of data. :(
files:scan --all
Does that get you "proper" files back?
Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!
This seems to actually be a bug in mod_fcgid used by FPM. It was fixed in version 2.3.10, but version 2.3.9 is still the only one proposed by Debian. I contacted the Debian maintainer and offered him a bounty to see if the new version could be uploaded to Debian.
In the meantime, NextCloud could implement an error message if this bug is detected, see this example:
// mod_fcgid <= 2.3.9 doesn't handle chunked transfer encoding for PUT requests
// see https://github.com/kd2org/picodav/issues/6
if (strstr($_SERVER['HTTP_TRANSFER_ENCODING'] ?? '', 'chunked') && PHP_SAPI == 'fpm-fcgi') {
// We can't seek here
// see https://github.com/php/php-src/issues/9441
$l = strlen(fread($stream, 1));
if ($l === 0) {
throw new Exception('This server cannot accept "Transfer-Encoding: chunked" uploads (please upgrade to mod_fcgid >= 2.3.10).', 500);
}
// reset stream
fseek($stream, 0, SEEK_SET);
}
For sysadmins, the only solution is to either compile your own mod_fcgid or to use mod_php instead.
@szaimen Completed how?
Sorry, closed by accident. Do you mind creating a PR with the proposed patch? Thanks a lot!
This is not a patch, but an idea, I'll let you implement it in NextCloud.
I fear I am not the right guy to implement this. However I suppose it would need to get added somewhere here: https://github.com/nextcloud/server/blob/9e73412e3f596d3bed478ecce4d014950d8993e9/apps/dav/lib/Connector/Sabre/File.php#L588 cc @nextcloud/server-backend
Problem occurs here too. Nextcloud 27.0.2, nginx/1.18.0 (Ubuntu) over FastCGI, PHP 8.1 (8.1.22-1+ubuntu22.04.1+deb.sury.org+1)
DAVx⁵ unfortunately has to use chunked transfer because it get its input data as stream, so the problem always occurs with DAVx⁵ (but also with curl when using chunked transfer).
Got the same problem with apache 2.4.57, mpm_event, mod_proxy_fcgi and php8.2-fpm on Debian 12. Temporary? solved by adding SetEnv proxy-sendcl
to the Apache configuration to tell mod_proxy to buffer the request and always send the Content-Length header. This may cause spooling requests to disk by apache, but there are no more 0 byte files.
Got the same problem with apache 2.4.57, mpm_event, mod_proxy_fcgi and php8.2-fpm on Debian 12. Temporary? solved by adding
SetEnv proxy-sendcl
to the Apache configuration to tell mod_proxy to buffer the request and always send the Content-Length header. This may cause spooling requests to disk by apache, but there are no more 0 byte files.
We use the same solution in the Nextcloud VM: https://github.com/nextcloud/vm/blob/master/lets-encrypt/activate-tls.sh#L124-L129
How can this still be a thing after 7 (!?!) years?
Ran into 0 byte video files when using chunked upload with PhotoSync app.
It seems to me everyone is busy developing new features. There are other issues like this ("Reliable Up/Download" #11138 for example) where I'm stumped that they were not found, fixed, tested and regression-tested since v1. I really hope the team is going to reverse course some day and develop a solid file-syncing solution.
We hit the annoying 0 byte files issue when using a
PUT
operation with a chunked transfer encoding to modify an existing 0 byte file. After successfully executing thePUT
the server replies with a 204 but the file is still empty. Not sure if this is a NC or sabre/dav issue.I'm using nginx 1.13.8 as a webserver (nginx 1.10.3 doesn't work either). My nginx configuration is quite default. The relevant part:
Some observations:
fastcgi_request_buffering off
tofastcgi_request_buffering on
solves the issue but this is not an option for large streamed files.Background: I'm a developer of Mountain Duck which uses the create/update file pattern described here http://sabre.io/dav/0bytes/. According this document nginx in a newer version should work fine. We have several users hitting this issue though.
Steps to reproduce
Expected behaviour
Files has updated content and correct size
Actual behaviour
Remote file is not touched at all and you still see 0 bytes
Server configuration
Operating system: Ubuntu 16
Web server: nginx
Database: MariaDB
PHP version: 7.0.22
Nextcloud version: (see Nextcloud admin page) 12.0.4
Updated from an older Nextcloud/ownCloud or fresh install: Fresh
Where did you install Nextcloud from: Official download