Chunked transfers lead to 0 byte files

ylangisc commented 6 years ago

We hit the annoying 0 byte files issue when using a PUT operation with a chunked transfer encoding to modify an existing 0 byte file. After successfully executing the PUT the server replies with a 204 but the file is still empty. Not sure if this is a NC or sabre/dav issue.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @report.csv "http://ubuntu.local/remote.php/webdav/ylatst.txt" -u admin:admin
*   Trying 192.168.48.135...
* TCP_NODELAY set
* Connected to ubuntu.local (192.168.48.135) port 80 (#0)
* Server auth using Basic with user 'admin'
> PUT /remote.php/webdav/ylatst.txt HTTP/1.1
> Host: ubuntu.local
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.54.0
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 204 No Content
< Server: nginx/1.13.8
< Date: Mon, 22 Jan 2018 16:41:33 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=xlmQsVcgE%2FcbE8%2FmB8bvsVG3fGrkDGB%2Fg5wHy5sJqrIjfeVotxGf4US%2BnLdGj6xNmzPjH3NzYgm09Qy%2FnkqKQA59LI9qQEmY0mkZZs17wwCvcubybbqiCS83FHYGdvhe; path=/; HttpOnly
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Set-Cookie: nc_sameSiteCookielax=true; path=/; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Content-Security-Policy: default-src 'none';
< Set-Cookie: oc4nuj3qa5xi=upmvsv2vtom8dg52tn72r3g4a0; path=/; HttpOnly
< Set-Cookie: cookie_test=test; expires=Mon, 22-Jan-2018 17:41:33 GMT; Max-Age=3600
< OC-FileId: 00000180oc4nuj3qa5xi
< ETag: "46ca66c845120b954836fb1dc6fb4af9"
< OC-ETag: "46ca66c845120b954836fb1dc6fb4af9"
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none

I'm using nginx 1.13.8 as a webserver (nginx 1.10.3 doesn't work either). My nginx configuration is quite default. The relevant part:

location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        fastcgi_pass      unix:/var/run/php/php7.0-fpm.sock;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_intercept_errors on;
        fastcgi_request_buffering off;
    }

Some observations:

changing fastcgi_request_buffering off to fastcgi_request_buffering on solves the issue but this is not an option for large streamed files.
it does happen for files larger than 7kb in my local environment. Smaller files work fine. For servers not being in my local network the files have to be even larger to show the issue. It's much more difficult to replicate the issue in such an environment. Due to bandwidth limitation and latency the server probably has enough time to process the data and a buffer overflow or similar is less likely to happen.

Background: I'm a developer of Mountain Duck which uses the create/update file pattern described here http://sabre.io/dav/0bytes/. According this document nginx in a newer version should work fine. We have several users hitting this issue though.

Steps to reproduce

Upload a 0 byte file
Try to update the file with chunked transfer encoding (see above for a curl example)

Expected behaviour

Files has updated content and correct size

Actual behaviour

Remote file is not touched at all and you still see 0 bytes

Server configuration

Operating system: Ubuntu 16

Web server: nginx

Database: MariaDB

PHP version: 7.0.22

Nextcloud version: (see Nextcloud admin page) 12.0.4

Updated from an older Nextcloud/ownCloud or fresh install: Fresh

Where did you install Nextcloud from: Official download

ylangisc commented 6 years ago

After some more testing I've noticed that it's even worse. It does not matter if you create a 0 byte file prior updating it. Basically all PUT operations with a chunked transfer encoding lead to 0 byte files in low latency and high bandwidth environments for files that are greater than a certain size (see above). In my local environment it's easy to replicate with a simple curl command. E.g.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @testfile.bin "http://ubuntu.local/remote.php/webdav/testfile.bin" -u admin:admin

nextcloud-bot commented 6 years ago

Hey, this issue has been closed because the label stale is set and there were no updates for 14 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

dkocher commented 6 years ago

Please reopen this issue.

Greek64 commented 5 years ago

I too am having this issue with webdav. After disabling encryption and sniffing the traffic going to and from the nextcloud webserver (nginx 1.14.0 in my case), I too came to the conclusion, that after a PUT command with chunked transfer encoding, nextcloud responses with 204 no content and creates only 0 byte files.

I noticed this behavior when trying to export my Kodi library via webdav to nextcloud.

I can also provide a sanitized HTTP communication if needed, but I guess that will not help much.

jospoortvliet commented 5 years ago

I just tested on nextcloud 15 with a 700 kb image. 256 kb of the image made it to the server. a 7.5 mb ODP file was also cut in half - 3.3 mb on the server.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @test.odp "http://127.0.0.1/nextcloud-15/remote.php/webdav/test.odp" -u jos:jos
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'jos'
> PUT /nextcloud-15/remote.php/webdav/test.odp HTTP/1.1
> Host: 127.0.0.1
> Authorization: Basic am9zOmpvcw==
> User-Agent: curl/7.62.0
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* Signaling end of chunked upload via terminating chunk.
< HTTP/1.1 201 Created
< Date: Thu, 03 Jan 2019 11:22:50 GMT
< Server: Apache
< X-Powered-By: PHP/7.3.0
< Set-Cookie: oczswzpgpvdq=9qbdh7bhvlq6s9um76vqfs7dlg; path=/nextcloud-15; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=I1gqnptmzWjmuBJhZqrfPOeNr6LIhTAn6OWTsFxQhflPr2EuxY2LtSmdoIMc0bz4ehfVuMBRmLO%2FZBBQG6FEDfgc0Q7qupNRWiHZPR%2BxESiNJg5Vh1%2Fajn6geMMRJNWv; path=/nextcloud-15; HttpOnly
< Content-Security-Policy: default-src 'none';
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: no-referrer
< Set-Cookie: nc_sameSiteCookielax=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Set-Cookie: oczswzpgpvdq=pj5kbp17hlf3btfob60d2meh7q; path=/nextcloud-15; HttpOnly
< Set-Cookie: cookie_test=test; expires=Thu, 03-Jan-2019 12:22:50 GMT; Max-Age=3600
< OC-FileId: 00001151oczswzpgpvdq
< Content-Length: 0
< ETag: "7ef58d6f933a4e3387fdf1418f188772"
< OC-ETag: "7ef58d6f933a4e3387fdf1418f188772"
< Content-Type: text/html; charset=UTF-8
< 
* Connection #0 to host 127.0.0.1 left intact

ldipenti commented 5 years ago

I just tested on nextcloud 15 with a 700 kb image. 256 kb of the image made it to the server. a 7.5 mb ODP file was also cut in half - 3.3 mb on the server.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @test.odp "http://127.0.0.1/nextcloud-15/remote.php/webdav/test.odp" -u jos:jos
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'jos'
> PUT /nextcloud-15/remote.php/webdav/test.odp HTTP/1.1
> Host: 127.0.0.1
> Authorization: Basic am9zOmpvcw==
> User-Agent: curl/7.62.0
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* Signaling end of chunked upload via terminating chunk.
< HTTP/1.1 201 Created
< Date: Thu, 03 Jan 2019 11:22:50 GMT
< Server: Apache
< X-Powered-By: PHP/7.3.0
< Set-Cookie: oczswzpgpvdq=9qbdh7bhvlq6s9um76vqfs7dlg; path=/nextcloud-15; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=I1gqnptmzWjmuBJhZqrfPOeNr6LIhTAn6OWTsFxQhflPr2EuxY2LtSmdoIMc0bz4ehfVuMBRmLO%2FZBBQG6FEDfgc0Q7qupNRWiHZPR%2BxESiNJg5Vh1%2Fajn6geMMRJNWv; path=/nextcloud-15; HttpOnly
< Content-Security-Policy: default-src 'none';
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: no-referrer
< Set-Cookie: nc_sameSiteCookielax=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Set-Cookie: oczswzpgpvdq=pj5kbp17hlf3btfob60d2meh7q; path=/nextcloud-15; HttpOnly
< Set-Cookie: cookie_test=test; expires=Thu, 03-Jan-2019 12:22:50 GMT; Max-Age=3600
< OC-FileId: 00001151oczswzpgpvdq
< Content-Length: 0
< ETag: "7ef58d6f933a4e3387fdf1418f188772"
< OC-ETag: "7ef58d6f933a4e3387fdf1418f188772"
< Content-Type: text/html; charset=UTF-8
< 
* Connection #0 to host 127.0.0.1 left intact

I have been doing some related tests on another webdav service with curl, and for me it started to work ok when using other Content-Type header, like "application/octet-stream" instead of "application/x-www-form-urlencoded".

rblayzor commented 5 years ago

Also running into this issue on Nextcloud 16.04. Nginx 1.16.1

macOS finder DAV client.

With "fastcgi_request_buffering off" uploads in Finder to the Nextcloud server over a DAV mount always end up "0 size".

If we enable "fastcgi_request_buffering on" it's hit or miss. Smaller files generally work, but larger files almost always end up truncated. (0 size).

Subito commented 4 years ago

Is this getting worked on? I would appreciate an update apart from the modified milestones. If I can help with debugging in any way (the first few posts do a good job on explaining how to reproduce the issue, IMHO), please let me know. Its quite an awful feeling that a part of my data gets destroyed once I use something like the iOS-App to Update files and it keeps me away from using anything other than the webinterface.

skjnldsv commented 4 years ago

@Subito sorry, not assigned yet.

tonihoang commented 4 years ago

I cannot push data from my QNAP with Hybrid Backup Sync, due to this, have not found any option to unchunk it yet.

lichtmetzger commented 4 years ago

This bug is still present with nginx 1.18.0.

With Apache + PHP-FPM, chunked transfers destroy all of my larger files very reliably. But that seems to be related to a different bug: https://bz.apache.org/bugzilla/show_bug.cgi?id=57087

So I've put nginx behind an Apache reverse proxy and then set these options in the nginx configuration so it doesn't error out on uploading larger files:

client_max_body_size 50000M;
client_body_buffer_size 400M;
fastcgi_buffers 64 4K;

I've tried uploading 30 files that are under <10k and half of them ended up being 0 bytes.

With fastcgi_request_buffering on only one time a file was truncated to 0 bytes in three subsequent test runs and very large files all transferred fine, too.

So while that option makes it a LOT better it is still not reliable and I cannot trust my data. mod_php being the only option to run Nextcloud reliably makes this very unsuitable for a VPS. It's already a bit painful that I have to run two web servers to make it almost work, but the blame on that can be put on the Apache team too, I guess.

I hope someone will fix this soon.

dkocher commented 4 years ago

We have documented our deployment workarounds in Zero byte file truncate issue with Nextcloud and ownCloud deployed with FastCGI.

j-dimension commented 4 years ago

Adding myself @j-dimension to the list of watchers. Happens to me with Nextcloud 18.06

stephenitpro commented 4 years ago

@dkocher thanks for the tip, I enabled this too on my Nginx Nextcloud config which improved the situation. Originally the documentation for Nextcloud calls for this fastcgi_request_buffering to be turned off. I had to turn it on to improve the situation since I'm waiting for NC19.0.1 to hopefully better address these type of issues. So much tinkering is required to make this setup work.

stephenitpro commented 4 years ago

Nevermind, I spoke too soon, it still is happening. The file uploads properly to the data directory but webdav/webui still report it as 0KB. The only option I have to run a files:scan --all but that takes forever for 50TB of data. :(

enoch85 commented 4 years ago

files:scan --all

Does that get you "proper" files back?

szaimen commented 1 year ago

Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!

bohwaz commented 1 year ago

This seems to actually be a bug in mod_fcgid used by FPM. It was fixed in version 2.3.10, but version 2.3.9 is still the only one proposed by Debian. I contacted the Debian maintainer and offered him a bounty to see if the new version could be uploaded to Debian.

In the meantime, NextCloud could implement an error message if this bug is detected, see this example:

        // mod_fcgid <= 2.3.9 doesn't handle chunked transfer encoding for PUT requests
        // see https://github.com/kd2org/picodav/issues/6
        if (strstr($_SERVER['HTTP_TRANSFER_ENCODING'] ?? '', 'chunked') && PHP_SAPI == 'fpm-fcgi') {
            // We can't seek here
            // see https://github.com/php/php-src/issues/9441
            $l = strlen(fread($stream, 1));

            if ($l === 0) {
                throw new Exception('This server cannot accept "Transfer-Encoding: chunked" uploads (please upgrade to mod_fcgid >= 2.3.10).', 500);
            }

            // reset stream
            fseek($stream, 0, SEEK_SET);
        }

For sysadmins, the only solution is to either compile your own mod_fcgid or to use mod_php instead.

bohwaz commented 1 year ago

@szaimen Completed how?

szaimen commented 1 year ago

Sorry, closed by accident. Do you mind creating a PR with the proposed patch? Thanks a lot!

bohwaz commented 1 year ago

This is not a patch, but an idea, I'll let you implement it in NextCloud.

szaimen commented 1 year ago

I fear I am not the right guy to implement this. However I suppose it would need to get added somewhere here: https://github.com/nextcloud/server/blob/9e73412e3f596d3bed478ecce4d014950d8993e9/apps/dav/lib/Connector/Sabre/File.php#L588 cc @nextcloud/server-backend

rfc2822 commented 1 year ago

Problem occurs here too. Nextcloud 27.0.2, nginx/1.18.0 (Ubuntu) over FastCGI, PHP 8.1 (8.1.22-1+ubuntu22.04.1+deb.sury.org+1)

DAVx⁵ unfortunately has to use chunked transfer because it get its input data as stream, so the problem always occurs with DAVx⁵ (but also with curl when using chunked transfer).

alx-tuilmenau commented 8 months ago

Got the same problem with apache 2.4.57, mpm_event, mod_proxy_fcgi and php8.2-fpm on Debian 12. Temporary? solved by adding SetEnv proxy-sendcl to the Apache configuration to tell mod_proxy to buffer the request and always send the Content-Length header. This may cause spooling requests to disk by apache, but there are no more 0 byte files.

enoch85 commented 8 months ago

Got the same problem with apache 2.4.57, mpm_event, mod_proxy_fcgi and php8.2-fpm on Debian 12. Temporary? solved by adding SetEnv proxy-sendcl to the Apache configuration to tell mod_proxy to buffer the request and always send the Content-Length header. This may cause spooling requests to disk by apache, but there are no more 0 byte files.

We use the same solution in the Nextcloud VM: https://github.com/nextcloud/vm/blob/master/lets-encrypt/activate-tls.sh#L124-L129

bcutter commented 2 months ago

How can this still be a thing after 7 (!?!) years?

Ran into 0 byte video files when using chunked upload with PhotoSync app.

Subito commented 2 months ago

It seems to me everyone is busy developing new features. There are other issues like this ("Reliable Up/Download" #11138 for example) where I'm stumped that they were not found, fixed, tested and regression-tested since v1. I really hope the team is going to reverse course some day and develop a solid file-syncing solution.

nextcloud / server