[Bug]: Error 499 while syncing thousands of new or modified files

kiv57 commented 5 days ago

⚠️ Before submitting, please verify the following: ⚠️

[x] This is a bug, not a question or a configuration issue.
[x] This issue is not already reported on Github (I've searched it).
[x] Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
[x] I agree to follow Nextcloud's Code of Conduct

Bug description

From the desktop client I added a local folder for syncing, containing about 50 000 files (photos).

After a few tens of seconds, the client shows a bunch of [\FilePath] : Network error : 499, then syncing speed falls to zero.

I can restart the sync : a hundred more files are synchronised before the errors appear again.

Steps to reproduce

From Windows desktop client, add a local folder to sync on server
Start syncing
Wait for about 100 files to sync and see errors on client.

Expected behavior

Smooth syncing of thousands of files.

Which files are affected by this bug

All local files (.jpg) waiting for syncing

Operating system

Windows

Which version of the operating system you are running.

Windows 11

Package

Official Windows MSI

Nextcloud Server version

30.0.2

Nextcloud Desktop Client version

3.14.3

Is this bug present after an update or on a fresh install?

Updated from a minor version (ex. 3.4.2 to 3.4.4)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

[x] Default internal user-backend
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Nextcloud Server logs

Hundreds of "File" is locked :

{"reqId":"l7HW7MVwWoeMkNZ5u5hG","level":3,"time":"2024-11-20T08:34:51+00:00","remoteAddr":"5.48.170.58","user":"vir","app":"no app in context","method":"POST","url":"/remote.php/dav/bulk","message":"\"Photos/2011/photos a reclasser 11642.jpg\" is locked","userAgent":"Mozilla/5.0 (Windows) mirall/3.14.3stable-Win64 (build 20241031) (Nextcloud, windows-10.0.26100 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"30.0.2.2","data":{"path":"/Photos/2011/photos a reclasser 11642.jpg"},"id":"673da21c06ae4"}
{"reqId":"l7HW7MVwWoeMkNZ5u5hG","level":3,"time":"2024-11-20T08:34:51+00:00","remoteAddr":"5.48.170.58","user":"vir","app":"no app in context","method":"POST","url":"/remote.php/dav/bulk","message":"\"Photos/2011/photos a reclasser 11641.jpg\" is locked","userAgent":"Mozilla/5.0 (Windows) mirall/3.14.3stable-Win64 (build 20241031) (Nextcloud, windows-10.0.26100 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"30.0.2.2","data":{"path":"/Photos/2011/photos a reclasser 11641.jpg"},"id":"673da21c06aed"}
{"reqId":"gyKijf5CLunGJGAZm6po","level":3,"time":"2024-11-20T08:34:51+00:00","remoteAddr":"5.48.170.58","user":"vir","app":"no app in context","method":"POST","url":"/remote.php/dav/bulk","message":"\"Photos/2011/photos a reclasser 11641.jpg\" is locked","userAgent":"Mozilla/5.0 (Windows) mirall/3.14.3stable-Win64 (build 20241031) (Nextcloud, windows-10.0.26100 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"30.0.2.2","data":{"path":"/Photos/2011/photos a reclasser 11641.jpg"},"id":"673da21c06af2"}
...

Additional info

Server setup :

Debian Bookworm
php-fpm 8.3
mariadb 11.5.2 (I left the 11.4 LTM version, maybe that was a mistake)

PHP config :

upload_max_filesize = 16G
post_max_size = 16G    
memory_limit = 16G   
max_file_uploads = 100
output_buffering = Off
max_input_time = 3600
max_execution_time = 3600
session.gc_maxlifetime = 3600
default_socket_timeout = 600 # changed from 60 with no change on bug

Edit : I tried to add OWNCLOUD_MAX_PARALLEL=1 to the client config file without success.

Darnokiv commented 2 days ago

I do have the same problem, but on different platforms as well (win10, win11, ubuntu24). The error started as 499, but changed to 299 by now. The symptoms are exactly the same, only the networkerror number shown changed. There are a few forum entries regarding this error from a few years ago, but there never was a concrete solution. Really appreciate @kiv57 opening the issue!

For me the syncing start to fail after the first 100 files as well, then it jumps to syncing file 200 and after this 400, 600, by then i usually restart the sync. Hope this helps identifiying the bug! 😃

Edit: The error just changed back to 499

kiv57 commented 2 days ago

I think I found a solution (also read my post in the help nextcloud forum).

Client log showed a 504 Gateway Time-out from the server, followed by exactly 100 files in error 499.

I tried several php and nginx settings, it appears to work after adding the following in my /etc/nginx/sites-available/cloud config file in the container, in the location ~ \.php(?:$|/) section :

fastcgi_send_timeout 600;
fastcgi_connect_timeout 600;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

Note that my NC instance runs in an incus container behind an nginx reverse proxy. I also added some of those parameters on the host side. After (tens of) hours of testing, I must admit that I haven't kept track of all my files changes.

Correct me if I'm wrong : I naively understand that the NC desktop client uploads the first 100 files one by one, then by packs of 100. The timeout occurs if the sync of a 100 pack files takes too long.

Edit : I will wait a bit to be sure that the bug is solved for me and Darnokiv before closing the bug.

Darnokiv commented 2 days ago

Thanks for the quick answer. Did you changed the config in the nextcloud container? My nextcloud runs as an app on truenas scale and I never got file editing inside a container working. I will try tomorrow, but right now I'm not sure what I would have to change exactly (within the nextcloud instance, the separate nginx reverse proxy on the server or the client). And since I don't have that much time free at the moment, this may take a while...

But thanks a lot for the help! I would really like to solve the problem in the long run.

kiv57 commented 2 days ago

I changed the mentioned settings in the host first (running the reverse proxy), then in the container.

The bug only disappeared when I added the lines to the container's nginx config, but I think the reverse proxy would also close the connection with default settings of 60 seconds. So you should change both sides if you can.

nextcloud / desktop