Files greater than 9.6Mb (10Mb) not syncing, but uploading to user UPLOADS directory

caos30 commented 2 years ago

Steps to reproduce

use client app (i tested Linux desktop and official android app)
in the client app define a local directory to be synchronized with a server user directory
put some files on that directory to be synchronized, some bigger than 9.3Mb and other smaller.

Expected behaviour

All filles should be synchronized and so appear on the server directory (i check it using web interface and also webdav from nautilus linux).

Actual behaviour

Only files smaller than 9.3Mb exists on this directory. In other words, the bigger than 9.3Mb files ARE NOT MOVED TO THERE. The linux desktop client shows a red message error like this for each one of those files:

Server replied: "404 Not Found" to "MOVE https://myNEXTCLOUDinstance.com/remote.php/dav/uploads/sergi/3493480664/.file"

Notes:

If i connect through SSH those "big" files are (chunked) on the directory data/{username}/uploads. It seems that the files pushed to server using remote.php API are first stored in that directory, in chunks about 10Mb. And i SUPPOSE that from there they are "joined" and moved to the user directory being synchronized. For some reason, those "big" files are there stored, but never become moved to its final destination. But it doesn't happen NEVER with small files, so? i would say that it's not a problem of file/directory permissions on the server side.
I have THE SAME PROBLEM trying to upload that "big" files through web browser (Files app). In this case i the message showed on screen after "upload" the file is:

Error when assembling chunks, status code 404

I think that it's more closer to the real problem than the error returned by the linux desktop app.

if i run ls -la data/sergi/uploads/ it returns:

drwxr-xr-x 32 myuser myuser 4096 Dec  4 18:59 .
drwxr-xr-x  7 myuser myuser 4096 Nov 27 11:45 ..
drwxr-xr-x  2 myuser myuser 4096 Dec  4 14:19 0ec4962a1e80a6c7d4f6f4d73754456c
drwxr-xr-x  2 myuser myuser 4096 Dec  3 12:02 2990573581
drwxr-xr-x  2 myuser myuser 4096 Dec  3 21:15 1480603860
drwxr-xr-x  2 myuser myuser 4096 Dec  3 12:04 1531137398
drwxr-xr-x  2 myuser myuser 4096 Dec  4 14:19 1b96ea58d9424626701d4971b49db0a3
...

and each one of this containg ONE OR MORE chunks, so if i list one of them i get something like this:

ls -la data/sergi/uploads/2990573581
total 16640
drwxr-xr-x  2 myuser myuser     4096 Dec  3 12:02 .
drwxr-xr-x 32 myuser myuser     4096 Dec  4 18:59 ..
-rw-r--r--  1 myuser myuser 10000000 Dec  3 11:55 0000000000000000
-rw-r--r--  1 myuser myuser  7025833 Dec  3 12:02 0000000000000001

Note: myuser is the server user managing the website, so all PHP/data files own to him. So i would say that it's not a problem of user permissions. In fact, the SMALL files are uploaded with errors. So, it's more a question of size. i suspect that the problem is clearly on the ASSEMBLING OF THE CHUNKS. Clearly the files not needing chunking in more than one chunk are not having problems. The problem seems to be when there are more than one chunk. !!!

What have i done/found?

I've checked all the usual limits for file sizes (apache, php, etc...), user permissions on server, etc... in fact i'm searching solutions during the last 4 days to this on official forum, and here on github. But i didn't get go far away thant see that it seems a problem with this "chunks asembling". I've tried even to read and understand the PHP files on server doing this assembling task, but sincerely i find this challenge away of my capabilities.

I've seen that this same problem was reported more than 3 years ago by other people, but or the issues where not "finished/solved" or the solutions there didn't work in my case.

I'm fully open to help you guys to find the "possible bug" during the next days. Please ask me what do you need i do. Thanks in advance!!

Server configuration

Operating system: Ubuntu 20.04, Linux 5.4.0-90-generic x86_64

Web server: Apache, NGINX, PHP-FPM

Database: MySQL 10.5.13

PHP version: 7.4.26

Nextcloud version: 22.2.3

Updated from an older Nextcloud/ownCloud or fresh install: fresh installation (i'm not 100% sure)

Where did you install Nextcloud from: i'm not quite sure... i would say that i downloaded a nuke PHP version, probably from official website or Github releases

Signing status:

Signing status

``` No errors have been found. ```

List of activated apps:

App list

``` Enabled: - accessibility: 1.8.0 - activity: 2.15.0 - breezedark: 22.1.0 - bruteforcesettings: 2.2.0 - calendar: 2.4.0 - circles: 22.1.1 - cloud_federation_api: 1.5.0 - comments: 1.12.0 - contacts: 4.0.6 - contactsinteraction: 1.3.0 - dashboard: 7.2.0 - dav: 1.19.0 - federatedfilesharing: 1.12.0 - federation: 1.12.0 - files: 1.17.0 - files_pdfviewer: 2.3.1 - files_rightclick: 1.1.0 - files_sharing: 1.14.0 - files_trashbin: 1.12.0 - files_versions: 1.15.0 - files_videoplayer: 1.11.0 - firstrunwizard: 2.11.0 - logreader: 2.7.0 - lookup_server_connector: 1.10.0 - mail: 1.10.5 - news: 16.2.1 - nextcloud_announcements: 1.11.0 - notes: 4.2.0 - notifications: 2.10.1 - oauth2: 1.10.0 - password_policy: 1.12.0 - passwords: 2021.11.20 - photos: 1.4.0 - privacy: 1.6.0 - provisioning_api: 1.12.0 - recommendations: 1.1.0 - serverinfo: 1.12.0 - settings: 1.4.0 - sharebymail: 1.12.0 - spreed: 12.1.2 - support: 1.5.0 - survey_client: 1.10.0 - systemtags: 1.12.0 - tasks: 0.14.2 - text: 3.3.0 - theming: 1.13.0 - twofactor_backupcodes: 1.11.0 - updatenotification: 1.12.0 - user_status: 1.2.0 - viewer: 1.6.0 - weather_status: 1.2.0 - workflowengine: 2.4.0 Disabled: - admin_audit - encryption - files_external - user_ldap ```

Nextcloud configuration:

Config report

``` $CONFIG = array ( 'instanceid' => 'ockgusu80mrd', 'passwordsalt' => '*****', 'secret' => '*****', 'trusted_domains' => array ( 0 => 'nc.mydomain.com', ), 'datadirectory' => '/home/myuser/web/nc.mydomain.com/public_html/data', 'dbtype' => 'mysql', 'version' => '22.2.3.0', 'overwrite.cli.url' => 'https://nc.mydomain.com', 'dbname' => '*****', 'dbhost' => '*****', 'dbport' => '', 'dbtableprefix' => 'oc_', 'mysql.utf8mb4' => true, 'dbuser' => '*****', 'dbpassword' => '*****', 'installed' => true, 'app_install_overwrite' => array ( 0 => 'tasks', 1 => 'bookmarks', ), 'maintenance' => false, 'theme' => '', 'loglevel' => 2, 'mail_smtpmode' => 'smtp', 'mail_smtpsecure' => 'ssl', 'mail_sendmailmode' => 'smtp', 'mail_from_address' => 'info', 'mail_domain' => 'mydomain.com', 'mail_smtpauthtype' => 'LOGIN', 'mail_smtpauth' => 1, 'mail_smtphost' => 'mydomain.com', 'mail_smtpport' => '465', 'mail_smtpname' => 'info@mydomain.com', 'mail_smtppassword' => '*****', ); ```

Are you using external storage, if yes which one: no.

Are you using encryption: i'm not very sure, how would i know it?

Are you using an external user-backend, if yes which one: no.

Client configuration

Browser: Firefox 94.04 (linux desktop)

Operating system: Ubuntu 20.04 with Gnome Shell

Logs

The most annoying thing is that i've not found errors on php/apache/nginx logs :-|

Desktop Client for Linux

Server replied: "404 Not Found" to "MOVE https://myNEXTCLOUDinstance.com/remote.php/dav/uploads/sergi/3493480664/.file"

Browser HTML popup error

Error when assembling chunks, status code 404

AASorokin commented 2 years ago

Desktop Client for Linux

Server replied: "404 Not Found" to "MOVE https://myNEXTCLOUDinstance.com/remote.php/dav/uploads/sergi/3493480664/.file"

Browser HTML popup error

Error when assembling chunks, status code 404

In nginx config for host: find location ~ /\. { deny all; } replace location ~ /\.(?!file).* { deny all; }

or (if using Let's Encrypt) find location ~ /\.(?!well-known\/) { deny all; return 404; } replace location ~ /\.(?!well-known\/|file) { deny all; return 404; }

caos30 commented 2 years ago

@AASorokin i love you man !!! It did the magic: edit the SSL nginx config file for host, in other words...

nano /home/myuser/conf/web/nc.mydomain.com/nginx.ssl.conf

and apply the change you suggested: replace

location ~ /\.(?!well-known\/) { deny all; return 404; }

by this other:

 location ~ /\.(?!well-known\/|file) { deny all; return 404; }

I want to emphasize that only works if you change the nginx.ssl.conf, hehehe... i've been wasting my time during several hours editing the file nginx.conf and making failure attempts to get it run.

Maybe for many of you -senior developers- it's obvious, but i was beginning to feel desperate when i searched other nginx files even apache2 conf files and .htaccess file (where indeed there is a 404 redirecting like this too!? but it seems that it doesn't affect). So finally, i make a file list of the nginx host configuration folder and i saw several other files nginx-----.conf

I hope this help other people. Indeed, i use HestiaCP on my VM to create and manage the hosts. I recommend a lot. A fully and wonderful opensource "sysadmin panel" (forked and legate of the veteran VestaCP).

My last question

It would be recommendable to change something in Nextcloud server code to avoid this problem to other people? Maybe a checking (and corresponding alert) on the "System" admin menu of the admin user (in Settings of the Nextcloud instance). I've seen that there appears sometims other "server settings alerts/recommendations" (cache, etc...).

Do you think i must open a new "feature request" ticket here? Thanks.

AASorokin commented 2 years ago

Maybe...

It would be more correct not to use hide files (the name starts with a dot: .file) in web requests. This is a mistake of the developers.

AASorokin commented 2 years ago

@AASorokin i love you man !!! It did the magic: edit the SSL nginx config file for host, in other words...
nano /home/myuser/conf/web/nc.mydomain.com/nginx.ssl.conf
and apply the change you suggested: replace
location ~ /\.(?!well-known\/) { deny all; return 404; }
by this other:
 location ~ /\.(?!well-known\/|file) { deny all; return 404; }

Update: Change this strings in templates: /usr/local/hestia/data/templates/web/nginx/default.tpl /usr/local/hestia/data/templates/web/nginx/default.stpl

caos30 commented 2 years ago

Update: Change this strings in templates: /usr/local/hestia/data/templates/web/nginx/default.tpl /usr/local/hestia/data/templates/web/nginx/default.stpl

Yes!! I'm sorry to not mention it in my previous message, because i certainly did so. I probably did it after write that previous message. Anyway, it can be useful for other people using HestiaCP arriving here. I add a new post on HestiaCP community forums precisely sharing this info, a couple of days ago.

In fact, i made a copy of those default.tpl and default.stpl NGINX templates (nextcloud.tpl and nextcloud.stpl) and there i did this modification of that exception rule. And after that, i modified at HestiaCP panel the web host where i've hosted Nextcloud to use this new NGINX template (it says Proxy template in the web form).

Cheers!

kesselb commented 2 years ago

Sounds like this issue is solved by changing the nginx configuration.

nextcloud / server