[Bug]: Folders deleted on Desktop Client are never deleted on the server with E2EE enabled - inconsistent sync state

bcutter commented 1 year ago

⚠️ Before submitting, please verify the following: ⚠️

[X] This is a bug, not a question or a configuration issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct

Bug description

Folders deleted on a Windows endpoint are only deleted locally, but never on the NC server as long as E2EE is activated in the Desktop Client.

It is for sure a leftover from https://github.com/nextcloud/desktop/issues/5564 where all that mess started.

Once E2EE is disabled (in the Desktop client itself), all previous performed deletions are performed instantly, so the "folder deletion queue" is being emptied. On the server they are removed, also the activity app confirms this.

Steps to reproduce

Enable E2EE (install app, activate on the Desktop client)
Create a folder (no matter if done on the web, another endpoint like iOS/Android client or the same Desktop Client, also doesn't matter if inside or outside an E2EE folder - right, it also affects folders outside of E2EE folders!)
Make sure the created folder exists on the server, maybe additonally check activity app
Delete a folder on the endpoint with Desktop client

Expected behavior

The folder is deleted locally and on the server.

Which files are affected by this bug

Every (!) folder, no matter if empty or not

Operating system

Windows

Which version of the operating system you are running.

Windows 10

Package

Appimage

Nextcloud Server version

27.0.1.2

Nextcloud Desktop Client version

3.9.0

Is this bug present after an update or on a fresh install?

Updated from a minor version (ex. 3.4.2 to 3.4.4)

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

Are you using an external user-backend?

[X] Default internal user-backend
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Nextcloud Server logs

Nothing in logs.

Additional info

https://github.com/nextcloud/desktop/issues/5564
Deletion of folders using other endpoints (the web interface, iOS client, Android client, ...) works without any issue, instantly.
Creation, renaming and moving folders is synced fine. It's "just" that deletion... which produces very very VERY confusing inconsinstent states over time, depending on the usage of NC very quickly.
Additionally (probably a 2nd issue or a variant of this one): folders INSIDE an E2EE folder which are deleted (e. g. using the iOS client), are not deleted on the Desktop client endpoint. E2EE is just a complete mess... 🙄🙄🙄

bcutter commented 1 year ago

Update:

Current workaround is to regularly (sometimes multiple times a day) disable E2EE on the endpoint, let NC sync everything (including deletion of folders) and re-enable E2EE with the not-so-handy passphrase again. A pain in the evening, needless to say.

Either correct sync or E2EE - both is not possible.

Tested this meanwhile on multiple Windows endpoint (Win10), one with NC desktop client v3.9.0, one with v3.9.1 - same results.

bcutter commented 1 year ago

Ahm, can we have ANY response please after two weeks?

Will 3.9.2 contain anything resolving this issue maybe?

bcutter commented 1 year ago

One more month without any response.

Even 3.9.4 made no change so far. Of course not, because nobody took care of this so far... 🤨

mgallien commented 1 year ago

the fact that a non empty e2e encrypted folder cannot be deleted is on purpose you cannot recover the deleted files as they are encrypted (i.e. server does not even know their names) hence this limitation

bcutter commented 1 year ago

WRONG!

@mgallien You didn't get the point here, please read the issue description.

Folders OUTSIDE (!!!) of E2E encrypted folders are affected.

The only connection to E2E is: once you disable it in the desktop client, the "queue" of folders to be deleted (from local 2 server) or which have been deleted (from server to client) is emptied, in other words:

E2E enabled in desktop client: normal folders can not be deleted ❌
E2E disabled in desktop client: normal folders can be/are deleted ✅

So reopen and let's work on that, finally.

bcutter commented 1 year ago

@mgallien See latest post. Reopen. Thanks.

bcutter commented 1 year ago

@mgallien Reminder 2.

bcutter commented 1 year ago

@mgallien reminder 3.

Do I need to open another issue for the very same bug?!? Come on.

bcutter commented 1 year ago

How can we fix this finally?!?

bcutter commented 1 year ago

@mgallien ?

bcutter commented 1 year ago

Ping @mgallien

joshtrichards commented 1 year ago

@bcutter Stop with the weekly/biweekly personal dev pings or I'll lock this issue. Thank you.

bcutter commented 1 year ago

Ok sir. So this means someone will finally have a look and fix this?

You'd be kind of motivated too if this bothers you on an every day basis creating REAL-LIFE problems.

Very frustrating to see absolutely ZERO progress after such a long time. Not even being asked to test or provide something. Really great. Sit and wait. Thank you.

bcutter commented 12 months ago

Tested with latest stable 3.10.1 on all Windows endpoints - no change. To sync deleted folders remotely/locally, one needs to

disable E2EE
let the client sync (deletions are now actually performed, even if they happened already days or weeks ago)
re-enable E2EE

Curious to see if 3.11.0 will change something. I doubt it.

bcutter commented 11 months ago

Same with 3.11.0. I wonder if

my NC is the only instance all over the world affected by this bug or if
others, especially those with one (Windows) endpoint only, are affected but simply never discover this (you need to compare web/server with the endpoint) and
if I could do or test or pay or provide anything on top of my current initial issue description to help devs.

At least 3) really could be answered, hopefully. I don't like the moderate to high probability this issue otherwise will end like https://github.com/nextcloud/end_to_end_encryption/issues/248#issuecomment-1879106724.

ph00lt0 commented 11 months ago

@bcutter honestly just going to tell you this isn't going to lead to anything. I used to be a naive Nextcloud fan boy basically since the company was founded. I had high hopes they would focus getting a stable FOSS solution. My frustrations are best described in tldr format here: https://discuss.privacyguides.net/t/remove-nextcloud/13907?u=ph00lt0. Just saying Nexcloud has always been like this and will stay a hobby project in the way that the current leadership is driving this car. I do feel sad about that.

bcutter commented 9 months ago

I want to share a fresh experience:

With NC Desktop client v3.12.0.20240213 (which does not contain anything related to this issue afaics according to https://github.com/nextcloud/desktop/releases/tag/v3.12.0) running since 2024-02-18, yesterday E2EE had an issue out of nowhere with adding or deleting files from one specific E2EE storage subfolder (started with one particular file - all other subfolders of E2EE encrypted storage and non-E2EE storage worked just fine), right after I had to go through the "disable E2EE locally temporarily to get other folders outside E2EE storage deleted, basically empty the deletion queue" workaround.

I then

backuped the somehow problematic subfolder locally on the Windows endpoint outside of NC synced storage
disabled E2EE on the Windows endpoint (all files got removed locally)
disabled E2EE app on the server
used the webinterface to delete that whole subfolder
then re-enabled the E2EE app on the server, re-enabled E2EE on the Windows endpoint, waited to complete the sync and re-added the formerly problematic folder - which synced fine, like all other folders too

I've gone through this "fix E2EE sync issues" in almost the very same way many times over the last years.

Today I accidentally discovered, that - with E2EE enabled (!) - folder deletions from outside E2EE folders, both locally and on the server, seem to sync just fine, meaning: folder deletions are performed instantly (normal expected behaviour, but... see the issue description). No clue why so far (I don't think it's the 3.12 release but rather the last "fix E2EE sync issue" process described above I went through) but for the moment for this single Windows endpoint it seems like the issue is not existing for now.

Unfortunately I did not have a look at the backend (NC log did not contain anything related, shown on warning level) like the lock tables in the database or sth. like that.

I will need to

a) monitor this behaviour on that single Windows endpoint for a while and
b) check what's the behaviour on other affected Windows endpoints (also "gone"? or just "fixed" for that single Windows endpoint?)

Update: I was excited a bit too early. Seems like there are fundamental issues with E2EE now, discovered when I tried to use it on another endpoint (iOS), see https://github.com/nextcloud/ios/issues/2809.

How can I start from scratch with E2EE? I need (or think it's worth a try) a complete reset, including everything on the server including private keys etc.

nextcloud / desktop