search

nextcloud / desktop

💻 Desktop sync client for Nextcloud
https://nextcloud.com/install/#install-clients
GNU General Public License v2.0
3.05k stars 801 forks source link

[Bug]: v3.8.0 basically KILLS E2EE ("Error with the metadata. Getting unexpected metadata format.") #5564

Open bcutter opened 1 year ago

bcutter commented 1 year ago

⚠️ Before submitting, please verify the following: ⚠️

Bug description

After updating to desktop client v3.8.0 I started to see strange sync issues. Encrypted changed and created files could not be synced with the server anymore.

With v3.8.0 there are metadata issues. Clients give Error with the metadata. Getting unexpected metadata format..

Steps to reproduce

  1. Start fresh with E2EE for a user by resetting E2EE (meta-data, private key, public key, potential file lock entries, corresponding filecache entries, delete all previously existing E2EE folders, run occ files:scan-app-data etc.)
  2. Desktop client with 3.8.0: enable E2EE by setting a passphrase
  3. Another client (I used the iOS app): clear E2EE from settings, supply same (new) passphrase as set in step 2
  4. Desktop client: create a new empty folder and encrypt it, put one file in it
  5. iOS client: try to access the encrypted folder

Note: you can also create the folder on the iOS client and try to access it from the desktop client, same result.

Expected behavior

Access is possible as usual.

Which files are affected by this bug

All E2EEncrypted!

Operating system

Windows

Which version of the operating system you are running.

Windows 10

Package

Appimage

Nextcloud Server version

25.0.5.1

Nextcloud Desktop Client version

3.8.0

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 3.3.6 to 3.4.0)

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

Are you using an external user-backend?

Nextcloud Server logs

This seems to be the initial issue (not fully sure):

{"reqId":"zCH6yCfRIeEVQuTq6QP7","level":3,"time":"2023-04-03T20:34:34+02:00","remoteAddr":"xxx.xxx.xxx.xxx","user":"Username","app":"no app in context","method":"DELETE","url":"/ocs/v2.php/apps/end_to_end_encryption/api/v1/lock/871525","message":"Intermediate meta-data file missing","userAgent":"Mozilla/5.0 (Windows) mirall/3.8.0stable-Win64 (build 20230331) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"25.0.5.1","exception":{"Exception":"OCA\\EndToEndEncryption\\Exceptions\\MissingMetaDataException","Message":"Intermediate meta-data file missing","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/end_to_end_encryption/lib/Controller/LockingController.php","line":133,"function":"saveIntermediateFile","class":"OCA\\EndToEndEncryption\\MetaDataStorage","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"unlockFolder","class":"OCA\\EndToEndEncryption\\Controller\\LockingController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/ocs/v1.php","line":63,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/ocs/v2.php","line":23,"args":["/var/www/nextcloud/ocs/v1.php"],"function":"require_once"}],"File":"/var/www/nextcloud/apps/end_to_end_encryption/lib/MetaDataStorage.php","Line":174,"CustomMessage":"--"}}

Nothing in logs for the client messages "Error with the metadata. Getting unexpected metadata format." !!!

Additional info

You should maybe have a look at what you did in e. g. (quick screening for E2E relevant changes)

I downgraded to 3.7.4 and everything is working as expected/normal! No more sync issues or error messages. Just a bit of data loss and loss of 4 hours... thanks.

Currently in rage mode and not willing to upgrade to 3.8.0 again to provide desktop client logs. Sorry.

mgallien commented 1 year ago

@bcutter sorry to hear that can you share a debug archive ? I can provide a secure share link for you to share it privately with us (and yes we have to keep this secret via a signed paper)

VanHell1638 commented 1 year ago

Hi, @mgallien i can provide the needed debug archive. Please send me the secure share link.

mgallien commented 1 year ago

@VanHell1638 I will delay this as I have found an issue in the end-to-end encryption implementation of the desktop client that is not conform to the specification I will work on a fix and will see if I can find a workaround to allow recovery from the bug very sorry for that

mgallien commented 1 year ago

just as a note, I did fix an issue with a missing part in the e2ee implementation of desktop client in 3.8.0 https://github.com/nextcloud/desktop/pull/5570 release is a work in progress https://github.com/nextcloud/desktop/issues/5569

mgallien commented 1 year ago

not sure how you can recover at least if the files are still correct and accessible on your client, doing any modification may be enough I will look into this after the release is done

bcutter commented 1 year ago

TBH after almost 5 lost hours yesterday I'm still not (yet) in the needed mood so providing debugs... better refer to @VanHell1638.

Just as note: I fixed this FOR ME by doing a full E2EE reset, downgrading desktop client (and blocking auto update on all other machines to stay at 3.7.4 for now) and re-syncing all E2EE content. Probably just downgrading would have been enough and the 3 to 4 hours of E2EE reset party was oversized.

I will test any new desktop client version claiming to have this fixed of course.

mgallien commented 1 year ago

https://github.com/nextcloud/desktop/pull/5572 should enable automatic recovery from broken checksums

mgallien commented 1 year ago

both fix will be in 3.8.1 release

bcutter commented 1 year ago

I will test 3.8.1 once it is released.

bcutter commented 1 year ago

Just a small note (not sure if it's worth posting @ https://github.com/nextcloud/end_to_end_encryption/issues) since updating the E2EE app from 1.11.1 to 1.11.3 yesterday:

  1. iPhone updated encrypted folder content (created new files)
  2. Desktop Client with v3.7.4 is in endless sync queue ("waiting", but never finishes)
  3. Disabling E2EE on the 3.7.4 desktop client made it sync other queued changes (outside of encrypted folders)
  4. Reenabling E2EE on the 3.7.4 desktop client starts syncing the encrypted folder content, but stucks and never finishes. grafik

Seems like 3.7.4 desktop client and 1.11.3 server app version don't work very well. Hopefully 1.11.3 will with 3.8.1... 🙄

Trouble continues. Had to disable E2EE on that client (and probably all other clients running the desktop client) to make the non-encrypted rest sync. Argh...

When will 3.8.1 gonna be released? Otherwise I would need to look into how to downgrade a server app (e2ee 1.11.3 to 1.11.1)...

bcutter commented 1 year ago

How to downgrade end_to_end_encryption server app? I'm going crazy here with stuck sync of really really REALLY important files ⚠

bcutter commented 1 year ago

Downgrading the server app back to 1.11.1 (theory: desktop clients back to 3.7.4, app back to 1.11.1 - combination where everything was simply WORKING) changed nothing. Still stuck sync:

initial sync after E2EE enabled on desktop client: grafik

restarted sync later: grafik

Had to upgrade server app back to 1.11.3 as iOS client refused changes (e. g. deleting E2EE files) with error message "Error - The server version of the end to end encryption is not compatible with this client". I guess some meta data generated with a newer server app version might cause this.

It's a complete mess, nothing is working anymore. I'm so upset. Especially cause it's like a déjà vu to what we saw back a few years ago when E2EE was originally marketed as production ready even it was (and is) technically still in beta according to the seriousness of still existing issues:

This is the marketing https://nextcloud.com/blog/desktop-3-8-end-to-end-encryption-levels-up-with-sharing-and-file-drop/, this #5564 is what's really going on. Just inacceptable, I should refuse and delay NC client updates for a few weeks in future, like proven for the server part. My mistake doing the update.

Need to cool down and hope some day someone will help on the 3.8.1 question (nightly, beta, hotfix, stable release) so I can stop praying I can finally sync again.

bcutter commented 1 year ago

After spending another 2 hours I am giving up now. Meanwhile desktop client only syncs encrypted files and complains about missing encryption information.

grafik

Congrats, you guys just really screwed up things. Out of abilites from a user perspective.

Wasted enough of my time. You'll fix it for sure. Just ping me once you wake up and have some news on this.

ckaspro commented 1 year ago

Awesome, just ran into the same trap. Even worse - a few days ago my laptop/ssd with the encrypted source files died - no access anymore.

Honestly: why is the 3.8.0 release still online? Can you imagine how many people are loosing (or not having access to - what is it?!) their files right now?

Can you please tell us:

thoniGH commented 1 year ago

I ran into the same issue a 6 days ago (Apr-08). My observations might help:

Jardo-51 commented 1 year ago

Also ran into this issue when upgrading desktop client from 3.7.4 to 3.8.0. Downgrading back to 3.7.4 didn't help.

I know this is free software and I appreciate it. But I also find it hard to comprehend why version 3.8.0 is still online with such a serious bug and why haven't you released the hotfix version when you had the problem fixed for 2 weeks already.

ghost commented 1 year ago

Same from my end. The error has been known for 2 weeks and according to the information here on Github and has already been solved. Where is the hot fix? As an alternative to Microsoft products, an urgent bug fix should not wait 2 weeks and a faulty version should still not still available for download. The quality assurance should be checked again at this point.

mgallien commented 1 year ago

please test again with the latest release it should fix the bugs in the current release and recover from broken metadata automatically https://github.com/nextcloud/desktop/releases/tag/v3.8.1

p5n commented 1 year ago

3.8.1 works better, but I removed .sync_XXXX.db from the folder, so it started working, but logs that encrypted files removed from server (maybe it tries to access them by not-encrypted path?) Android app also broken, not sure if it is related: https://github.com/nextcloud/android/issues/11536

timebuzzerfelix commented 1 year ago

first try of 3.8.1 looked good, but then we discovered that thousands of local files have now a size of 0 KB. We also installed the 3.8.1 client on a new machine and made a sync, same problem: files are shown with file name but have 0 KB. So we lost mostly of our local data. We really hope that the encrypted files are still on the server.

mgallien commented 1 year ago

3.8.1 works better, but I removed .sync_XXXX.db from the folder, so it started working, but logs that encrypted files removed from server (maybe it tries to access them by not-encrypted path?) Android app also broken, not sure if it is related: nextcloud/android#11536

if you use virtual files on windows, removing the sync database is going to lead to more troubles the client will be very confused by the state and may take wrong decisions

mgallien commented 1 year ago

first try of 3.8.1 looked good, but then we discovered that thousands of local files have now a size of 0 KB. We also installed the 3.8.1 client on a new machine and made a sync, same problem: files are shown with file name but have 0 KB. So we lost mostly of our local data. We really hope that the encrypted files are still on the server.

if it possible for you @timebuzzerfelix please send us a debug archive covering the sync after you upgraded I can provide a secure file drop link to privately share it

timebuzzerfelix commented 1 year ago

Short Update: the iOS app also shows the files with 0KB. So maybe this bug is also related to the recent E2EE Update on server side?

ghost commented 1 year ago

Client 3.8.1 freezes on Linux and macOS as soon as it wants to synchronise. Under Linux, you can only shut down the process (Flatpak App as well as Debian Package based App) + 100% CPU load. Under macOS, the beach ball of death appears. In my case, nothing works at all. Since I also use it for work, this is a considerable disruption to my business.

bcutter commented 1 year ago

3.8.1 on Windows (10) working fine in terms of no freezes or similar, but: related to E2EE, it still can't download all available encrypted content (only 2 out of 5 sub folders, example for one encrypted folder):

grafik

Update:

ckaspro commented 1 year ago

So far (Windows 3.8.1), I am waiting since 1h to be able to get the Client opened - it is unresponsive. I will report further in hope that it actually does something (I have that on 2 Windows- PCs...).

bcutter commented 1 year ago

2nd affected Windows desktop client also fine (fixed) after 3.8.1 update. No sync issues anymore, no new freeze issues like others reported. OK from my side. (note: I did not yet perform the restart the installer asks for at the end of the update)

mgallien commented 1 year ago

2nd affected Windows desktop client also fine (fixed) after 3.8.1 update. No sync issues anymore, no new freeze issues like others reported. OK from my side. (note: I did not yet perform the restart the installer asks for at the end of the update)

the restart is in fact only needed when we change the integration into the files explorer but there is not really a way to condition the restart in a smart way that would tell the user when it is really needed and when not

mgallien commented 1 year ago

anybody with freezes could you share logs ? I would assume something may appear in the logs to help with the analysis (I cannot reproduce this) I can provide a secure file drop link (i.e. the files are encrypted and shared securely with me and I have an obligation to keep them private)

ckaspro commented 1 year ago

So far (Windows 3.8.1), I am waiting since 1h to be able to get the Client opened - it is unresponsive. I will report further in hope that it actually does something (I have that on 2 Windows- PCs...).

Unfortunately no further success, both stuck since days under Windows 10 (after several restarts). That's it for me (no more time to wait for a solution), have to switch to another solution :/ Thanks anyways for your support!

ghost commented 1 year ago

Matthieu, Thank you for your efforts.. Please let us know which logs can help you.

To support your request I tried under Linux (Pop!_OS): kill the client and then start it via "nextlcoud --logwindows" in Terminal. Unfortunatelly the generated folder /tmp/Nextcloud-logdir is totally empty. Even after half an hour. Oh by the way, the client was of course started and of course hung up again straight away.

Regards Michael

mgallien commented 1 year ago

Matthieu, Thank you for your efforts.. Please let us know which logs can help you.

To support your request I tried under Linux (Pop!_OS): kill the client and then start it via "nextlcoud --logwindows" in Terminal. Unfortunatelly the generated folder /tmp/Nextcloud-logdir is totally empty. Even after half an hour. Oh by the way, the client was of course started and of course hung up again straight away.

Regards Michael

@MichlFranken you can use nextcloud --logfile test.log

jingojango commented 1 year ago

sync without encryption (uncheck encrypted directory) works fine. But 3.7.4 downloads the encrypted files again but put them in folders like "0cb6e...ebf....". 3.8.0 and 3.8.1 hangs if encrypted folder is enabled. 3.8.1 hangs with log:

2023-04-25 20:19:16:517 [ info nextcloud.sync.clientsideencryption /home/user/src/libsync/clientsideencryption.cpp:549 ]:       decryptStringSymmetric key:  "..."
2023-04-25 20:19:16:517 [ info nextcloud.sync.clientsideencryption /home/user/src/libsync/clientsideencryption.cpp:550 ]:       decryptStringSymmetric data:  "..."
2023-04-25 20:19:16:517 [ info nextcloud.sync.clientsideencryption /home/user/src/libsync/clientsideencryption.cpp:97 ]:        found parts: ("...", "...") old format? false
2023-04-25 20:19:16:517 [ info nextcloud.sync.clientsideencryption /home/user/src/libsync/clientsideencryption.cpp:561 ]:       decryptStringSymmetric cipherTXT:  "..."
2023-04-25 20:19:16:517 [ info nextcloud.sync.clientsideencryption /home/user/src/libsync/clientsideencryption.cpp:562 ]:       decryptStringSymmetric IV:  "..."
2023-04-25 20:19:16:517 [ debug nextcloud.metadata /home/user/src/libsync/clientsideencryption.cpp:1639 ]       [ OCC::FolderMetadata::setupExistingMetadata ]: encrypted file "real filename" "..." "..."

3.8.1 seems to use more cpu than 3.8.0. 3.8.1 is even not responsible any more (need to run 3.8.1 to get to settings to disable crypted folder). For 3.8.1 i'am using the appImage. It seems that 3.8.1 hangs on the file where 3.8.0 reported an error during the first synchronization attempts. Therefore, could there still be erroneous partial data left somewhere in the cache that are now causing problems?

OS: Linux

Update: It seems that the new Nextcloud has some more problems. It could not longer sync files with change date 1970. After fixing the dates with the script from https://blog.admin-intelligence.de/nextcloud-synchronisierung-meldet-ungueltige-aenderungszeit-quickfix/ the client 3.8.0 thought it is a good idea to delete all encrypted files. Hopefully i backuped all and did not loose a file. But after creating a new folder, encrypt it an copy all files there the sync with 3.8.1 is the same. Some files are uploaded but at some point the sync hangs. So the idea that some cache files would be the problem is probably false.

timebuzzerfelix commented 1 year ago

In my case after installing 3.8.1. the files were synced but many with size 0KB. The next day it started syncing again, downloaded GB of data, and all files were local available and in correct size. I was happy, but after few hours, the same files had 0KB again. So it seems to be in kind of a loop. On the iOS mobile app the files are still shown with 0KB. I can not figure out what kind of files are synced with 0KB, file type or last change date seems not the criteria.

mgallien commented 1 year ago

sync without encryption (uncheck encrypted directory) works fine. But 3.7.4 downloads the encrypted files again but put them in folders like "0cb6e...ebf....". 3.8.0 and 3.8.1 hangs if encrypted folder is enabled. 3.8.1 hangs with log:

2023-04-25 20:19:16:517 [ info nextcloud.sync.clientsideencryption /home/user/src/libsync/clientsideencryption.cpp:549 ]:       decryptStringSymmetric key:  "..."
2023-04-25 20:19:16:517 [ info nextcloud.sync.clientsideencryption /home/user/src/libsync/clientsideencryption.cpp:550 ]:       decryptStringSymmetric data:  "..."
2023-04-25 20:19:16:517 [ info nextcloud.sync.clientsideencryption /home/user/src/libsync/clientsideencryption.cpp:97 ]:        found parts: ("...", "...") old format? false
2023-04-25 20:19:16:517 [ info nextcloud.sync.clientsideencryption /home/user/src/libsync/clientsideencryption.cpp:561 ]:       decryptStringSymmetric cipherTXT:  "..."
2023-04-25 20:19:16:517 [ info nextcloud.sync.clientsideencryption /home/user/src/libsync/clientsideencryption.cpp:562 ]:       decryptStringSymmetric IV:  "..."
2023-04-25 20:19:16:517 [ debug nextcloud.metadata /home/user/src/libsync/clientsideencryption.cpp:1639 ]       [ OCC::FolderMetadata::setupExistingMetadata ]: encrypted file "real filename" "..." "..."

3.8.1 seems to use more cpu than 3.8.0. 3.8.1 is even not responsible any more (need to run 3.8.1 to get to settings to disable crypted folder). For 3.8.1 i'am using the appImage. It seems that 3.8.1 hangs on the file where 3.8.0 reported an error during the first synchronization attempts. Therefore, could there still be erroneous partial data left somewhere in the cache that are now causing problems?

OS: Linux

@jingojango I will do a custom build with extra log around the code to recover from the bug in 3.8.0 to try to figure out the exact issue will ping back here

bcutter commented 1 year ago

Today's experience:

  1. Synced few E2EE content using iOS app
  2. Added one document on a Windows machine while NC desktop client being offline
  3. Started NC desktop client on that machine later
  4. Sync stuck.
  5. Checked NC log: 
    Fehler  no app in context   OCA\EndToEndEncryption\Exceptions\MissingMetaDataException: Intermediate meta-data file missing 
Fehler  no app in context   OCA\EndToEndEncryption\Exceptions\MissingMetaDataException: Intermediate meta-data file missing 
Fehler  no app in context   OCA\EndToEndEncryption\Exceptions\MissingMetaDataException: Intermediate meta-data file missing
  6. Checked nc_e2e_encryption_lock which contained one entry with timestamp 1682584095 which seems to match the document change time (- 1 hour, maybe due to timezone delta?) according to https://www.unixtime.de/index.php
  7. Removed the document from NC, deleted that lock entry, re-added that document to E2EE folder - sync worked, document accessible on other E2EE enabled clients (iOS app), nc_e2e_encryption_lock table remained empty.

Maybe there's some kind of issue when doing changes to E2EE folders (at least adding content) when NC client is offline? Just a wild guess.

Interestingly there's a delta of 15 files which are never synced: grafik (changed 1 file)

grafik (changed nothing but forced sync)

...and NC log still gives few

Fehler  no app in context   OCA\EndToEndEncryption\Exceptions\MissingMetaDataException: Intermediate meta-data file missing 
vor Sekunden
Fehler  no app in context   OCA\EndToEndEncryption\Exceptions\MissingMetaDataException: Intermediate meta-data file missing 
vor 2 Minuten
Fehler  no app in context   OCA\EndToEndEncryption\Exceptions\MissingMetaDataException: Intermediate meta-data file missing
vor 6 Minuten

Update:

Desktop client made changes not available on the server so iOS client is blind? Did deleting the e2ee file lock table entry cause this?

Never-ending story hmm. What's broken now again? Was working really fine for days now without any issues running 3.8.1.

ghost commented 1 year ago

Is there an update on this in the meantime? For some of us, syncing Nextcloud (with E2EE) hasn't worked for weeks, which is quite unpleasant.

Thanks

bcutter commented 1 year ago

For me it is working... somehow.

bcutter commented 1 year ago

Note: https://github.com/nextcloud/desktop/pull/5657 is part of https://github.com/nextcloud/desktop/releases/tag/v3.8.2 stable release.

timebuzzerfelix commented 1 year ago

still problems, lot of files seems to get synced, but all have size of 0KB. removing and adding again the folder from syncing does not help. but files seems to be still on the server in correct size.

ghost commented 1 year ago

Unfortunately it still doesn't work for me. Although the app no longer hangs and reacts to clicks, no files are synchronized. It calculates the changes and then it shows a number of files to sync but it's stuck at 0 out of 8234 and has been for days. So yes, the freezing has been fixed and no, unfortunately the sync with activated E2E still does not work for me.

p5n commented 1 year ago

3.8.2 looks still broken for me too.

Did not look into the sources but my guess is that app enables encryption too late, so it messes plain file names with encrypted file names.

Sometimes encrypted names appear in the list, sometimes it gives error that plain file name "could not be located" on the server.

mgallien commented 1 year ago

@bcutter my understanding is that this is now resolved in the 3.9.0 latest release so far I would like to close it to not clutter the open issues

bcutter commented 1 year ago

@bcutter my understanding is that this is now resolved in the 3.9.0 latest release so far I would like to close it to not clutter the open issues

@mgallien it looked like it was fixed, correct. Unfortunately, after a longer time I experienced a follow-up issue probably of the fixes: folders deleted on the endpoint with NC desktop client are not deleted on the server with E2EE enabled. Please urgently have a look at https://github.com/nextcloud/desktop/issues/5918. Thank you.

github-actions[bot] commented 1 year ago

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

bcutter commented 1 year ago

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

@mgallien see last post, thx.

github-actions[bot] commented 1 year ago

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

bcutter commented 1 year ago

Well since this issue, https://github.com/nextcloud/desktop/issues/5918 exists which has been closed based on false assumptions by @mgallien .

Would love to see every mess being cleaned up instead of being quickly closed for no reason.

Reopen and READ https://github.com/nextcloud/desktop/issues/5918 ! Thanks.

github-actions[bot] commented 1 year ago

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

rklasen commented 1 year ago

I was affected by this too. I enabled E2EE for two folders only (the most important ones naturally), and ever since I accessed them with the mobile app sometime around April or May I can't access them anymore.

When they get downloaded to the PC, the folder names are some hex values instead of the real file names.

Seeing how this is supposedly fixed now, is there any way I can recover my lost folders? I have the 12 word mnemonic of course.

EDIT:

This is what the client looks like for me. The mnemonic is set up, but all filesnames are messed up, most of the folders are empty and files are missing from the folders that aren't empty.

image