GuyPaddock
commented
4 years ago @CorruPTo what web server are you using? What type of PHP (Apache or FPM)?
Closed dronesportugal closed 8 months ago
GuyPaddock
commented
4 years ago @CorruPTo what web server are you using? What type of PHP (Apache or FPM)?
dronesportugal
commented
4 years ago @CorruPTo what web server are you using? What type of PHP (Apache or FPM)?
PHP FPM 7.2 Nextcloud 17.0.1 stable
Hope to answer everything you asked.
GuyPaddock
commented
4 years ago @CorruPTo We were seeing this too... because of the size of the file involved, nginx is timing-out before PHP is done assembling the final file.
With the current version of Nextcloud, you would want to adjust your NGinx configuration to increase timeouts to give NC more time to assemble files after upload:
fastcgi_connect_timeout 60;
fastcgi_send_timeout 1800;
fastcgi_read_timeout 1800;This allows up to 30 mins of time to pass. It probably doesn't need to be quite that high, but it all depends on how fast your server disk I/O is.
It'd be great if a future version of Nextcloud had a daemon to handle this kind of task out-of-proc (like Seafile does), since larger files are becoming pretty normal.
dronesportugal
commented
4 years ago @CorruPTo We were seeing this too... because of the size of the file involved, nginx is timing-out before PHP is done assembling the final file.
With the current version of Nextcloud, you would want to adjust your NGinx configuration to increase timeouts to give NC more time to assemble files after upload:
fastcgi_connect_timeout 60; fastcgi_send_timeout 1800; fastcgi_read_timeout 1800;This allows up to 30 mins of time to pass. It probably doesn't need to be quite that high, but it all depends on how fast your server disk I/O is.
It'd be great if a future version of Nextcloud had a daemon to handle this kind of task out-of-proc (like Seafile does), since larger files are becoming pretty normal.
Can you let me know the location of the Nginx configuration? its not on the nextcloud folder. Maybe on /etc/php/7.2/fpm? wish file?
GuyPaddock
commented
4 years ago @CorruPTo It depends on what nginx containers you are using... In our case, it's located in /etc/nginx/nginx.conf
zarmhast
commented
3 years ago Issue still reproducible in Nextcloud 20.0
I cannot set nginx/PHP-fpm timeouts to high values, as it only shifts the problem, instead of solving it, and at some point I start getting network disconnects instead of 504.
My proposed solution would be to mark files as ready to be reassembled, and handle the lengthy reassembly process in a cron task (cron.php). It seems a simple enough change, yet resulting in a way more stable file server. Especially considering that it is not that easy to add a daemon to a PHP-FPM server.
burntoc
commented
3 years ago Same problem here. Frustrating this has been lingering so long.
SMH17
commented
3 years ago It returns this error with big files upload even when actually they are correctly uploaded (tested checksum, no integrity issues, after upload in processing task it is triggered timeout event while uploaded file chucks are in PHP temp folder with assembling still in progress). This needs to be addressed, Nextcloud should keep connection alive till the chucks are fully assembled, preventing these timeout events and related error. Setting huge timeouts isn't a good practice.
Configurations where I have reproduced this issue
[1] OS: Debian Buster (ARM 64bit) PHP: 7.3 Apache: 2.4 Nextcloud: 20.0.4
File size uploaded with this error 12GB, 33GB Upload mode: Web interface drag-and-drop (Chromium browser 87.0.4280.88)
[2] OS: Debian Buster (x86-64) PHP: 7.4 Nginx: 1.18 Nextcloud: 20.0.4
File size uploaded with this error 40GB Upload mode: Web interface drag-and-drop (Chromium browser 87.0.4280.88)
reavessm
commented
3 years ago Still happens on Nextcloud 20.06 and Pgsql 11.9. I set the fastcgi settings but that still didn't fix it
brtptrs
commented
3 years ago After adjusting the timeouts we have the situation that big uploads end with "Error when assembling chunks, status code 504" errors only to have the files appear "magically" in nextcloud after some time. It seems that die the feedback on screen can also be misleading.
SMH17
commented
3 years ago The issue is still here even with the latest Nextcloud 21.
kiplingw
commented
3 years ago I confirm this with 21.0.1.1. I hope we won't have to upgrade to version 22 in order to have this fixed, since some of us on shared hosting don't have access to the newer MySQL required to run it.
voyager-hr
commented
3 years ago Did anyone manage to find solution for this?
In short I can upload, it seems, any size file, but when it says processing files "it fails with that 504 error" but file IS there and matches checksum of original file.
I thought it might be chunks size (default to 10MB) - I coudn't for some reason get it changed to different value so I changed code in PHP to 100 1024 1024 instead of default 10 1024 1024. But problem still persists.
Any other ideas? I tried now adding that timeout parameter but I somehow doubt that is the reason because I tested it is QUITE close to 2.5GB wehere it fails for me ... so 2.3GB I can upload no problem (either on LAN 1G speed or 10mbit adsl line) but 2.6 fails (that is why I thought maybe it can handle only 256 or close chunks).
I am running it inside docker container using SSD as storage and image linuxserver/nextcloud:amd64-version-21.0.1
voyager-hr
commented
3 years ago OK I think I have resolved this after hours of debugging - when I got desperate and posted comment above and resumed investigation of the issue.
I tried mounting upload and destination directories separatly as ramdisk (tmpfs), then both, but benefit was neglible.
So after realizing it is some kind of timeout and using my kitchen timer found it at 60 sec.
I found this to resolve my issue:
In nc container I edited /config/nginx/site-confs default site config for nginx and ADDED (just bold part: fastcgi_read_timeout 3600s;):
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $path_info;
fastcgi_param HTTPS on;
**# 20210523 Dorijan - default is 60s and assembling uploaded chunks takes longer than 60 seconds
fastcgi_read_timeout 3600s;**
fastcgi_param modHeadersAvailable true; # Avoid sending the security headers twice
fastcgi_param front_controller_active true; # Enable pretty urls
fastcgi_pass php-handler;@voyager-hr This is a bug that must be fixed Nextcloud side, keeping connection alive till the completion, rather than increasing timeout. As I have said above, using huge timeout in order to workaround the issue isn't a good practice.
voyager-hr
commented
3 years ago @voyager-hr This is a bug that must be fixed Nextcloud side, keeping connection alive till the completion, rather than increasing timeout. As I have said above, using huge timeout in order to workaround the issue isn't a good practice.
Does anyone from nextcloud team even pay attention to this issue/are they aware of it?
szaimen
commented
3 years ago Is this Issue still valid in NC21.0.3? If not, please close this issue. Thanks! :)
SMH17
commented
3 years ago The issue is still present in the latest 21.0.3
szaimen
commented
3 years ago If you are using Nginx, did you make sure to configure it correctly? https://docs.nextcloud.com/server/stable/admin_manual/installation/nginx.html?highlight=nginx
SMH17
commented
3 years ago The server is configured correctly and the bug is caused by the fact Nextcloud doesn't keep connection alive till the completion, so when a very large file is uploaded during the time needed to assembly all chunks it goes in timeout.
szaimen
commented
3 years ago Hm... I am not sure if this is fixable by us since it looks like it runs into a timeout issue from the webserver side if it takes too long.
cc @nextcloud/server-triage for more input on this.
ghost
commented
3 years ago This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.
SMH17
commented
3 years ago The issue is still here also on latest 22.1.0
pierreozoux
commented
3 years ago Ok, we did spend substantial time to investigate this issue. We think we have somehow the same issue, but maybe it is different.
The behavior is that if in the web UI we uploade a file is > 1GB, then we get in the UI, this error file is locked, existing lock on file: exclusive✖
If we reload the page, the file is actually fine.
In the network tab of firefox, we get a 423 Locked status code on the move request:
MOVE | https://nuage.beta.liiib.re/remote.php/dav/uploads/admin/web-file-upload-761e5de13b91cf0c4ab6f34b6164e593-1629271135519/.file
This MOVE request is actually spending almost exactly 1min in the waiting state. So it looks like a timeout somewhere.
We run our tests on the 20.0.11 which the latest 20.x at the time of writing.
And we could reproduce this behavior on 3 distinct instances, running on 2 different kubernetes cluster, but going to the same ceph rados gateway (ObjectStore) cluster. (The issue is not related to s3)
The upload takes less than 1hour, so we could rule out this issue.
This issue looks really similar to https://github.com/nextcloud/server/issues/19223, I'll close it as a duplicate of this one.
It'd be great if a future version of Nextcloud had a daemon to handle this kind of task out-of-proc (like Seafile does), since larger files are becoming pretty normal.
Yes, We think in the general sense, this comment is a good recommendation to solve it in general.
But for people that use s3, we think that this feature request is a better idea.
If there is anything we can do to help further, let us know :) In the mean time, we'll tell our users to not upload files bigger than 1GB on the web UI, and either spilt the file, or use a client to uplaod bigger files.
juliushaertl
commented
3 years ago 504 is a gateway timeout that would hint to too low settings for timeouts on the webserver process. Please check https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/big_file_upload_configuration.html for hints on the configuration values or ask in the forum for help on the webserver configuration. I'll close this though since if there is a gateway timeout happening there is not much we can do form the server perspective.
As a sidenote, there is some ongoing work to improve the performance of the final move request for chunked uploads on s3 in https://github.com/nextcloud/server/pull/27034 in the future.
@pierreozoux The locking issue seems rather unrelated. Please open a separate one for that. For additional debugging information you could actually enable debug logging for file locks as per https://github.com/nextcloud/server/blob/f14b8aa34bfb9f7af7b23cfecc09e2fb8f604c1b/config/config.sample.php#L1832-L1840 in order to see where the lock is happening before.
SMH17
commented
3 years ago @juliushaertl This issue shouldn't be closed because it's caused by poor implementation, not by server misconfiguration. Connection should be maintained alive till the end of the file upload processing, instead Nextcloud uses a simple http and after the upload completion doesn't keep alive the connection anymore during the time needed to assemble chunks and this triggers the timeout. Using huge timeout leads to performance problems and it's a bad practice. In addition, with many third party CDN services without the possibility to set big timeout on CDN, not even a huge timeout on the host server could solve the problem, because if the connection hasn't any response in a reasonable time, a timeout will be triggered CDN side regardless host settings. It's enough that Nextcloud maintains the connection alive and this could be done in many ways, e.g., abstracting upload activity and using Inter-Process Communication or using Web Sockets, HTTP/2 streams or any transport protocol that give you more control, also sending immediately new requests in Long polling till the end of the upload would work.
solracsf
commented
3 years ago Another option would be to allow to completely disable chunked uploading (from GUI, or from clients) with a simple config.php option, and fallback to regular (full-file) uploads.
We all know all the benefits of chunked uploading (resume aborted uploads is one of them, among others) but I believe administrators should have that option if they don't want/can't deal with these problems.
BartAgterbosch
commented
2 years ago Issue still present in v22.2, doesn't seem it will ever get fixed
GuyPaddock
commented
2 years ago @juliushaertl You guys are dodging the issue. Please consider re-opening to address with an architectural solution to this. Other software like Seafile and FileRun are able to handle this without excessive timeouts or special server configuration.
juliushaertl
commented
2 years ago Ok, let me reopen that than for the general discussion around that for non-s3 related fixes.
kiplingw
commented
2 years ago I agree. I am surprised this has taken so many years to fix. For digital media projects it's not unusual for users to need to upload and share raw video that could be several gigabytes in size. In those situations NextCloud is still unusable.
mikperk
commented
2 years ago Same f*cking issue here. Error appears after a 2GB file upload. I tried everything on the internet but nothing helped. Sad.
hoangnamitc
commented
2 years ago I had a similar problem. Here is my htaccess config. However still not working when I upload file size ~5GB
php_value memory_limit 2G
php_value upload_max_filesize 16G
php_value post_max_size 16G
php_value max_input_time 3600
php_value max_execution_time 3600Last week, I had the issue again. I have s3 provided by minio.
One of the Nextcloud log:
message":{"Exception":"Sabre\\DAV\\Exception","Message":"An exception occurred while uploading parts to a multipart upload. The following parts had errors:\n- Part 6: Error executing \"UploadPart\" on \"https://minio/bucket/urn%3Aoid%3A2097524?partNumber=6&uploadId=xxx\"; AWS HTTP error: cURL error 55: OpenSSL SSL_write: Broken pipe, errno 32 (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://minio/bucket/urn%3Aoid%3A2097524?partNumber=6&uploadId=xxx (server): 100 Continue - \n"On the nginx in front of minio, I got this log:
xxx - - [18/Feb/2022:15:54:05 +0000] "PUT /bucket/urn%3Aoid%3A2097642?partNumber=6&uploadId=xxxx HTTP/1.1" 408 25 "-" "aws-sdk-php/3.171.21 GuzzleHttp/7" 690 59.999 [minio-hot-minio-443] [] - - - - xxWith the 408, this is when I started to think, it might come from Nc itself. And the time 59.999 looks like a timeout somewhere.
I searched the s3 aws-sdk-php and Nc code, and finally found the option to fix it on my side:
https://lab.libreho.st/libre.sh/docker/nextcloud/-/commit/ad85b9cfc4106366375366a86b97c03497405cde
I added uploadPartSize to the s3 configuration. This is a temp workaround, as the real s3 multipart is coming, but I though it might help some people in the meantime.
grickard
commented
2 years ago I am facing the same issue with NC 23.0.2 My NC instances sit behind HAProxy. Even after increasing the time-out in my HAProxy front-end I am still facing this issue.
BartAgterbosch
commented
2 years ago Okay so to anyone that is running nextcloud as a docker container behind a reverse proxy, I highly suggest (as painful as it may be), to switch to the linuxserver/nextcloud container instead of the default nextcloud container, it will run on port 443 by default, and I haven't had a single problem since.. In fact I'd say it's even snappier out of the box than the default container and seems to have many issues plaguing the standard nc container fixed..
The only painful part about it ofc is you'll have to deploy it next to the old nc container, and move all the data from that old container to the volumes of the new container, then chown all that crap since you'll move from running as user 33 to 1000.
But I must say since I have moved to the linuxserver/nextcloud container, I couldn't be happier, no more issues whatsoever thusfar. I suppose overwriting http to https with the nextcloud container running on port 80 and not seeing what is going on outside of its' own container is just a recipe for issues
SMH17
commented
2 years ago @BartAgterbosch As I have said before, this issue is caused by the fact Nextcloud doesn't keep connection alive after the upload phase ends and start the chunks processing completion, so when a very large file is uploaded during the time needed to assembly all chunks it goes in timeout. Changing host isn't a solution. They need to use something more adequate to handle upload.
BartAgterbosch
commented
2 years ago @SMH17 I suppose so, but then I don't know if perhaps the linuxserver container is modified in any way, but after switching to that image instead, I haven't run into any issue since, perhaps they fixed it, and the official nextcloud image hasn't yet.. All I can say so far with certainty, for the people that wish to spare themselves the headache of trudging through hundreds of settings trying to fix what the nextcloud team apparently doesn't want to, the linuxserver version of the image doesn't seem to have any of these issues.
I don't know what it is they did that fixed it (only obvious difference I see is the port, probably some settings too then), but their version does not have the issues the native nextcloud image does have, don't know why, I'm not an expert, all I know is it works.
Instead of ignoring this glaring issue that has been around for years now, the nextcloud dev team might take a page out of the linuxserver image's book, whatever it is they did to fix it
grickard
commented
2 years ago @BartAgterbosch My setup utilizes vhosts and not dockers.
zubanst
commented
2 years ago Well, I have this issue too, (see the https://github.com/nextcloud/server/issues/31939 closed as duplicate of this one). What I don't get is this: if the file is correctly uploaded eventually (in my environment it is, cos I checked the checksum) then this message Error when assembling chunks, status code 504 is a kind of false positive brought up by nextcloud core (although reported by app server components), so why don't just get rid of if the file assembly is done after all? If the file truly cannot be uploaded, there would be a way to popup a specific upload error message based on this failure, right ?
cyberjohn60
commented
2 years ago I have been having the problem since upgrading from 22 to 23. After searching online and trying different setting changes I could not upload large files (in this case 11.1 GB) without timing out after 30 minutes and even having issue with smaller files. I finally got to it work and I think the issue is with the PHP7.4 configuration. Also believe there is disconnect between the way Nextcloud and PHP handle uploads. After during a deep dive into the issue, here are my observation and fix.
My setup as follows: Ubuntu Server 20.04.3 PHP7.4-FPM Apache/2.4.53 (Ubuntu) Nextcloud 23.0.3
PHP Configyration: allow_url_fopen = On file_uploads = On memory_limit 512M upload_max_filesize 16G post _max_size 16G max_input_time 3600 max_execution_time 3600
My Network is a 1 GIG upload and download.
I create a folder to place the temporary files here /var/big_temp_file/ and notice that during the upload process the data going to the folder was very small compare to the settings I have in place.
I remember there has been a few update recently with PHP7.4 so I believe PHP is not reading the setting correctly related to the upload_max_filesize and post _max_size. So the changes I made to get it to work are as follows.
In Nextcloud config.php I add the following configuration; 'remember_login_cookie_lifetime' => '1800', 'check_for_working_htaccess' => true, 'tempdirectory' => '/var/big_temp_file/'
In PHP7.4, I change the following to megabyte instead of using GB. post_max_size = 16000M upload_max_filesize = 16000M upload_tmp_dir = /var/big_temp_file/
I reboot the server after making the changes and now the amount of data going into the folder has increase. It is now only taking 3-5 minutes for a 3.5 GB file and it took 12-14 minutes for the 11.1 GB file. Has of now there are no more 'Error when assembling chunks, status code 504' and no more timeouts are being observed. Hopefully this information will help the developer with pinpointing the root cause of the issue.
YoitoFes
commented
2 years ago I'm running nextcloud as a container behind nginx reverse proxy, and increasing nginx's proxy_read_time resolved 504 error. Container image is official one (23.0.4-apache).
hoangnamitc
commented
2 years ago How to resolution this issues on Nextcloud Docker apache? Tks!
xom4ek
commented
2 years ago .htaccess
php_value memory_limit 2500M
php_value upload_max_filesize 16000M
php_value post_max_size 16000M
php_value max_input_time 3600
php_value max_execution_time 3600
php_value upload_tmp_dir /tmp/@cyberjohn57.php
<?php
$CONFIG = array (
'remember_login_cookie_lifetime' => '1800',
'check_for_working_htaccess' => true,
'tempdirectory' => '/tmp/'
);/tmp - big pvc mount
Work with s3 as backend and files ~7GBs
mrAceT
commented
2 years ago As of version 24.0.3 this is still an issue.
All the suggestions above have been implemented and the 504 error remains. Also the files do get uploaded (after some time)!
There is no difference between a "web upload" or the "desktop app", both give an error and both do process the file.
szaimen
commented
1 year ago Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!
zarmhast
commented
1 year ago Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!
Since about 23, don't remember the version, it is no longer possible to upload large files because float is not integer.
Because of that inability to handle large files any longer and because it is marked as "won't fix" I have long uninstalled nextcloud. IMHO, due to its inability to handle large files at all (runtime type exception when handling file size), the chunking issue is no longer relevant.
szaimen
commented
1 year ago Handling large files is not a problem at all with e.g. https://github.com/nextcloud/all-in-one. So it is really only a configuration issue when it doesn't work in my experience.
BlueH2O
commented
1 year ago Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!
I am on 25.0.2 and the issue persists.
szaimen
commented
1 year ago Die you follow all the steps of https://docs.nextcloud.com/server/stable/admin_manual/configuration_files/big_file_upload_configuration.html to enable large file uploads?
How to use GitHub
Steps to reproduce
Expected behaviour
Tell us what should happen Expected file to be upload without errors
Actual behaviour
Tell us what happens instead Error 504 assembling chunks and file is locked, unable to move or delete
Server configuration
Operating system: Ubuntu 18.04 Web server:
Database: Type: pgsql
Version: PostgreSQL 10.10 (Ubuntu 10.10-0ubuntu0.18.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0, 64-bit
Size: 14.4 MB PHP version: PHP Version: 7.2.24
Memory Limit: 512 MB
Max Execution Time: 3600
Upload max size: 1000 MB Nextcloud version: (see Nextcloud admin page)
Updated from an older Nextcloud/ownCloud or fresh install:
Where did you install Nextcloud from: Professionaly install Daniel Hansson T&M Hansson IT AB https://www.hanssonit.se
and installed from Unraid app, 2 different sources, same issue Signing status:
Signing status
``` Login as admin user into your Nextcloud and access http://example.com/index.php/settings/integrity/failed paste the results here. ```List of activated apps:
App list
``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your Nextcloud installation folder ```Nextcloud configuration:
Config report
``` If you have access to your command line run e.g.: sudo -u www-data php occ config:list system from within your Nextcloud installation folder or Insert your config.php content here. Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …) ```Are you using external storage, if yes which one: local/smb/sftp/...
Are you using encryption: yes/no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
LDAP configuration (delete this part if not used)
LDAP config
``` With access to your command line run e.g.: sudo -u www-data php occ ldap:show-config from within your Nextcloud installation folder Without access to your command line download the data/owncloud.db to your local computer or access your SQL server remotely and run the select query: SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap'; Eventually replace sensitive data as the name/IP-address of your LDAP server or groups. ```Client configuration
Browser: Google Chrome latest version Operating system: Windows 10
Logs
Web server error log
Web server error log
``` Insert your webserver log here ```Nextcloud log (data/nextcloud.log)
Nextcloud log
``` Insert your Nextcloud log here ```Browser log
Browser log
``` Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ... ```