Closed melroy89 closed 1 month ago
Same issue, got 2TB of files on server and local side. All synchronization fails after i updated the desktop client. On my other computer, i did not update the client, and it remains functional.
Also, the main dialog would not open on the desktop client, i can only see the error messages in the settings page.
This seems to apply to Mac OS X Desktop client also. As my customer reports same issue, while they haven't yet downgraded the client, I have urged them to do so. I don't have a Mac myself so I can't verify.
On windows 11, with all latest updates, the desktop client simply crashes now with 3.14, 3.13.3, 3.13.4 complaining about: Error path for program: C:\Program Files\Nextcloud\nextcloud.exe Module error path: C:\WINDOWS\SYSTEM32\ntdll.dll
I have the same issue with windows client and there are no errors in server log. The client is able to synchronize one or two files and then it aborts. After Downgrading to client v3.13.4 the synchronization is working again.
@mgallien this is an urgent call for action due to a high impact and regression issue with the latest desktop client.
I have same problem. I cannot upload files larger than 10 MB via Windows (11) desktop client - ver 3.14. Uploading via web interface works fine.
I have the same issue with windows client and there are no errors in server log. The client is able to synchronize one or two files and then it aborts. After Downgrading to client v3.13.4 the synchronization is working again.
I have the same problem. Error log desktop client v3.14 "GOAWAY received, cannot start a request"
Yup, same issue here on Ubuntu Desktop. Nextcloud_sync.log
only says Connection closed
for the failed items while the sync window shows that error in addition to the Server is unable to maintain the header compression....
error.
Oh god! I was diagnosing my server all day! I just found this thread... On linux, all versions of 3.14.0 are borked! (Flatpak, repo, and Appimage) All files larger than a few hundred MBs are causing issues... I tried all the tricks under the sun (chunks, proxy buffer, everything on the Nextcloud doc and elsewhere)
Turns-out this release is simply broken!
I can't give more context than what was already given... reverting to 3.13.[...] fixes it!
same here on MacOS 15.0
Same here. Server version 30 with latest windows client, using VFS on windows 11 (latest updates as well). First the "connection close" message on a few select files, and now I get spammed with "Conflict when uploading a folder. It's going to get cleared" messages, NC renders the filesystem very unstable as well.
Reverting to 3.13.4 fixed the issue.
I suggest pulling release 3.14 completely until a fix is found. As I am managing several dozens of clients (including their install), having to revert the NC install on every one of them if they click the update button is just NOT an option.
This should be marked as urgent and a priority fix.
EDIT : It would appear that something about charsets has been borked between 3.13 and 3.14, as the first file that NC complained with the "connection closed" had an "é" character in its name (and the sync stopped failing when I renamed it).
Now, after reverting the client to 3.13.4, some paths have charset issues. This was quickly fixed by re-configuring the sync.
The folders themselves did not get renamed, only the sync in the client.
Oh god! I was diagnosing my server all day! I just found this thread... On linux, all versions of 3.14.0 are borked! (Flatpak, repo, and Appimage) All files larger than a few hundred MBs are causing issues... I tried all the tricks under the sun (chunks, proxy buffer, everything on the Nextcloud doc and elsewhere)
Turns-out this release is simply broken!
I can't give more context than what was already given... reverting to 3.13.[...] fixes it!
Yes, I was also debugging this issue myself late in the night for hours. Until I reverted back a desktop client version, the error messages were very misleading as if the server was the problem (spoiler: the server is not the issue)!
You are right, all releases of 3.14.0 are borked under Linux. It's insane they even shipped this release. Did they even test it? It's even more insane nobody from Nextcloud responded yet to this ticket. It's a huge regression issue, large impact and high likelihood. Nextcloud didn't do anything for days now. The best thing they could do for now is pull back the release (or if they have a quick fix, fix it asap).
For anybody reading this, please go to your Desktop client where the error occur(s)/occurred and click at Setting -> General -> "Create Debug Archive" and upload the file here.
I'm not a nextcloud developer.. but I might blame potentially one of these PRs (or multiple): https://github.com/nextcloud/desktop/pull/6986 or: https://github.com/nextcloud/desktop/pull/6965, or: https://github.com/nextcloud/desktop/pull/6588 or even: https://github.com/nextcloud/desktop/pull/6987. Which feels like potential candidates of the root-cause of this issue.
I am seeing the same thing. But I can add that there are lines like this in the webserver access log:
myurl:443 myip - - [21/Sep/2024:13:53:20 +0200] "PROPFIND /remote.php/dav/uploads/username/3333697971 HTTP/2.0" 404 1199 "-" "Mozilla/5.0 (Macintosh) mirall/3.14.0daily (Nextcloud, macos-23.6.0 ClientArchitecture: arm64 OsArchitecture: arm64)"
and corresponding lines in nextcloud log,
webdav -- NotFound File with name //number could not be located
I don't know if this helps.
I will just repeat myself now..
For anybody reading this, please go to your Desktop client where the error occur(s)/occurred and click at Setting -> General -> "Create Debug Archive" and upload the file here.
Yep they need to just re-release 13.13.4 as a .1 to this version, been days now
Experiencing this too
Same issue with windows client. Downgrading to client v3.13.3 the synchronization is working again.
Shame the logs don't show much of anything besides the "connection closed" message.
Same problem for me, after updating to 3.14 none of my files would synchronize, even after deleting the target folder and all app data (in .config, .cache and .local/share) and completely setting up the application from scratch. Here the debug archive (It's a bit too big for Github, so I just shared it over my Nextcloud)
After downgrading to 3.13.4 everything worked again. I'm using NixOS and the Nix package.
Debug-Archiv Windows-Client: Download... I hope this helps with troubleshooting.
I'm not a nextcloud developer.. but I might blame potentially one of these PRs (or multiple): #6986 or: #6965, or: #6588 or even: #6987. Which feels like potential candidates of the root-cause of this issue.
I believe I was correct... It's this PR that caused the issue: https://github.com/nextcloud/desktop/pull/6986
And I notice that the change is reverted in this PR (and just recently merged. aka 1 hour ago): https://github.com/nextcloud/desktop/pull/7182
I will update you, but I'm not part of the Nextcloud dev team.
Same here.. Worked by reverting, but apart from people coming here... it's quiet sad for all the rest with auto-update who are stuck at the moment. Good luck!
Unfortunately, Nextcloud has not yet withdrawn the update or released a patch. We have decided to temporarily disable http/2 because the clients update almost automatically and our users get confused. Perhaps this suggestion will help others who have no control over the clients.
I am sorry for that, but I will call-out devs who contributed in the last few weeks to get a bit of attention on this major issue that impacts operations for many! @mgallien @claucambra the latest version (3.14.0) is broken on all OS. The issue seems to be tied to the way special characters are managed, but I don't have the knowledge to confirm. Reverting to previous versions 100% fixes the issue.
Symptoms are server error messages that are in fact not server errors, but client errors ("connexion closed", "unauthorized", "server responded [blank]", etc.) The server does not even have info level errors in the log.
If you are not the person that can deliver an emergency patch, could you please find who can? This issue makes Nextcloud client completely unusable at the moment, and is delivered as an auto-update for many user!
@nextcloud-bot
(If calling devs here is not accepted, please let me know, and I will remove that message... I think we are just getting a little desperate for a word from the dev team to get our users back on track!)
This issue has been referenced in the #7191 bugs overview macro-issue and the underlying bug has been reverted and backported to 3.14.
It has been noticed and is being worked on.
This issue has been referenced in the #7191 bugs overview macro-issue and the underlying bug has been reverted and backported to 3.14.
It has been noticed and is being worked on.
Thanks a lot for the precisions.
Can we safely update to V3.14 now or is the patch not deployed yet? (I still see V 3.14.0 as the current version) We are on the Flatpak version BTW.
Thank you.
Has not been re-released yet, but there are CI builds you can try if you are using appimages. https://github.com/nextcloud-desktop-bot/ci-builds/releases/tag/PR-7182
Not sure it's all fixed tho, so wait for a release I guess.
...we need a way to make the server tell the client to skip or force an update.
@Webbeh there are also daily builds for linux and windows https://download.nextcloud.com/desktop/daily/
Good to know, thanks!
Same here. Enterprise production server and home lab server have the same behavior as the client (v. 3.14.0) when syncing many small files or even a few large files
@Webbeh there are also daily builds for linux and windows https://download.nextcloud.com/desktop/daily/
I understand that, but the issue is that I can't install a daily build on all my user's machines, just to reinstall a production version in a couple days from now... The course of action here should be to release an emergency patch (probably consisting of reverting to the previous version, but naming it 3.14.1) on the main release channel...
Nextcloud is a commercial product now, aiming to compete with G-Suit and O365, and should be treated as such!
(I don't mean to sound harsh here, that is good news, it simply implies some changes in philosophy!)
Edit: And don't get me wrong, Microsoft releases broken code every other week to production, impacting all the users... But they are super quick when it comes time to revert a broken patch... I am not a fan of O365, but credit goes where credit is due!
I can confirm that I have exactly the same behavior using Windows 11. If I'm not mistaken mine was showing Connection closed
instead of Operation Canceled
. After I downgraded to v3.13.4
it started working just fine. I even tested using virtual file system and normal sync.
I really don't understand how something like this could be released. Any common end-to-end test should immediately flag such a breaking issue. You need to step up your quality control.
I really don't understand how something like this could be released. Any common end-to-end test should immediately flag such a breaking issue. You need to step up your quality control.
I don't think shaming is the solution, I am sure they did tests that did not include the exact situation we are in... "$h!t happens" as we say! But my true concern is that an emergency patch/revert has not been released on public channel yet! This situation will result in unnecessary support to users as the cause is known... Just revert while you fix it!
Don't get me wrong, I really appreciate the work done, especially considering it's completely free, but maybe more beta testing should be done before release, it just creates unnecessary emotion and stress for everyone :).
@bendschs The "free" argument doesn't fit exactly here because Nextcloud's development is funded by enterprise contracts. So this sort of thing should never happen.
@bendschs The "free" argument doesn't fit exactly here because Nextcloud's development is funded by enterprise contracts. So this sort of thing should never happen.
sure, just saying that i am probably not in the position to complain as i am not a paying customer. i agree though, that this just should not happen.
hello
we have one notable fix missing from this test build (https://github.com/nextcloud/desktop/pull/7200)
I really need your help with testing this in your environment
so please if you can test it ?
this is a build from stable branch that will become 3.14.1
https://cloud.nextcloud.com/s/QcjPFiwJaWJbqeH
with the extra fix for wrong timeout (only if you get this error message: Connection timed out
)
https://cloud.nextcloud.com/s/WM7kZiHHgXjnCxQ
The first link gives a build with the GUI being extremely slow to respond (same as with the 3.14 official release) and it does not sync the files. I can't move the window around easily. The sync starts but doesn't give an error and is just stuck at the initial "synchronizing" phase.
Took 10 minutes to click the "create debug archive" button and it just freezes explorer. Note that I am syncing the "Documents" windows folder with Nextcloud (where NC offers to save the debug archive at first) and I do NOT have onedrive installed so they don't eat each other.
Basically this is the same issue as 3.14 before the "connection timeout" message happens.
EDIT : also note that I rebooted the computer as prompted by the installer.
Exactly the same behaviour with the second link but seems more responsive, I actually managed to create a debug archive.
I am not expecting it to show much to be honest, besides the millions of lines like those :
discovery.cpp:1405 ]: Not a move, no item in db with inode 207858
discovery.cpp:1385 ]: Wiping virtual file without db entry for "<filepath>"
usermodel.cpp:683 ]: Item "Documents" retrieved resulted in "Conflict when uploading a file. It's going to get removed!"
No clue what those mean but I definitely don't have this in the 3.13.4 instance.
Across all the sync folders, I have about 3TB of files synced spread across 48000 folders and 355500 files (according to occ files:scan
), so I am expecting massive slowness if the sync goes awry.
Furthermore, after reverting back to 3.13.4 again, I got the same issue as last time : the charset of the folder syncs are wrong again. I guess it gets corrupted between 3.13.4 and 3.14.
Sorry for the multiple posts.
EDIT: @mgallien You guys are welcome to teamviewer request me if you want to see it in action on this computer. Anytime tomorrow is fine by me (swiss time).
the charset of the folder syncs are wrong again.
I guess that's #7188.
Most likely.
I wonder how much of an impact #7188 has on this issue, potentially resolving that one will help with this one?
They're both tracked in the same issue so let's see.
hello we have one notable fix missing from this test build (#7200) I really need your help with testing this in your environment so please if you can test it ? this is a build from stable branch that will become 3.14.1 https://cloud.nextcloud.com/s/QcjPFiwJaWJbqeH with the extra fix for wrong timeout (only if you get this error message:
Connection timed out
) https://cloud.nextcloud.com/s/WM7kZiHHgXjnCxQ
I cannot test because I am on Linux and these are windows install..
But please, for the love of god, @mgallien, can you contact whoever is in charge of versioning so that they create a version 3.14.1 based on the latest working 3.13.x, in an emergency, so that I stop getting call from users who have received a broken auto-update...
We really don't care about the features of 3.14 for now as it just does not synchronize, which is the main purpose of the Nextcloud client. This solution would give the team time to test and fix the underlying issue...
I am sorry to sound so negative, but this is really poor crisis management from the Nextcloud team! After an update, you get hundreds of issue tickets that the client stopped working... instead of continuing to receive new tickets, just revert!!! I have no idea who to talk to about this at Nextcloud, but this is major!
EDIT: I have just spotted the 3.14.1 release, but the flatpak version is not up to date yet... Is that something that can be discussed here or is the flatpak version maintained elsewhere?
Is 3.14.1 the same as 3.13.4 in terms of this bug? Has everything been reverted to sync properly?
It is not the same as 3.13.4 but it fixes the problem for me
this is now solved with the release of 3.14.1
thanks to whoever made this happen 🙏
So this error doesn't happen again. However, now I am having Conflict when uploading a file. It's going to get removed!
for what appears to be EVERY SINGLE FILE in my sync folders, and the GUI is extremely slow and unresponsive.
I made sure the sync tasks were 100% done before updating to the release 3.14.1, and now everything is uploading again.
I don't know what you guys changed in 3.14 regarding VFS but something is definitely broken, and those releases need much more testing and reviewing before shipping, especially since I mentioned there were still issues in my comments 3 days ago (a few posts above here on this very issue) and there was NO feedback on my comments.
Putting this here as I honestly don't want to revive issue #3912 or #4636 when those errors never happened until 3.14. This is all related.
Please reopen.
Also link to the debug archive since it's too big for github : https://www.swisstransfer.com/d/c2494e8c-518c-451b-a65e-5847ad10dce4
⚠️ Before submitting, please verify the following: ⚠️
Bug description
We got weird "operation canceled" errors according to the Nextcloud Desktop App.. While I do NOT see these kind of errors on the server side.
Going back to a older version (like desktop client 3.13.3) directly synced, no errors, no issues! And yes using the same server. Meaning we have a regression issue here (especially when there are large amount of files and folders on the server and client, the latest desktop client just fails hard).
Server side Nginx access log (no errors in the error log of Nginx), and even no errors in Nextcloud logs, which actually seems fine:
Also NO related errors in the Nextcloud log either. The errors you do see in the nextcloud app seems to be unrelated I believe.
The same sync errors are happening on both the Deb PPA as well as the AppImage. Meaning it doesn't matter which way you download the latest desktop client, it's not a packaging issue.
Steps to reproduce
Expected behavior
I do not expect regression with regard to the latest desktop client version (3.14.0). While I do not experience any issues by downgrading to 3.13.3 (v3.13.4 should also work).... After I downgraded and I now upgrade again to 3.14.0 and do NOT see the issues anymore..
Which doesn't mean that 3.14.0 is stable, it just means that whatever the error was, 3.13.3 and 3.13.4 could resolve the problem while 3.14.0 could NOT.
Which files are affected by this bug
/Algemeen (directories)
Operating system
Linux
Which version of the operating system you are running.
Linux Mint 21.3
Package
Official Linux AppImage
Nextcloud Server version
30.0.0
Nextcloud Desktop Client version
3.14.0
Is this bug present after an update or on a fresh install?
Updated from a minor version (ex. 3.4.2 to 3.4.4)
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
Are you using an external user-backend?
Nextcloud Server logs
Additional info
Related tickets: https://github.com/nextcloud/desktop/issues/5356 (it's an old ticket, while now I can pin-point exactly the issue between two desktop client version causing regression).
I'm happy to share the archive debug log, but not in public. I hope you understand.