[Bug]: v3.8.0 basically KILLS E2EE ("Error with the metadata. Getting unexpected metadata format.")

bcutter commented 1 year ago

⚠️ Before submitting, please verify the following: ⚠️

[X] This is a bug, not a question or a configuration issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct

Bug description

After updating to desktop client v3.8.0 I started to see strange sync issues. Encrypted changed and created files could not be synced with the server anymore.

I thought "well, just that time of the year E2EE had its hiccups, just start over" and spent almost 4 hours in resetting E2EE.
In the end it did fix nothing.

With v3.8.0 there are metadata issues. Clients give Error with the metadata. Getting unexpected metadata format..

Steps to reproduce

Start fresh with E2EE for a user by resetting E2EE (meta-data, private key, public key, potential file lock entries, corresponding filecache entries, delete all previously existing E2EE folders, run occ files:scan-app-data etc.)
Desktop client with 3.8.0: enable E2EE by setting a passphrase
Another client (I used the iOS app): clear E2EE from settings, supply same (new) passphrase as set in step 2
Desktop client: create a new empty folder and encrypt it, put one file in it
iOS client: try to access the encrypted folder

Note: you can also create the folder on the iOS client and try to access it from the desktop client, same result.

Expected behavior

Access is possible as usual.

Which files are affected by this bug

All E2EEncrypted!

Operating system

Windows

Which version of the operating system you are running.

Windows 10

Package

Appimage

Nextcloud Server version

25.0.5.1

Nextcloud Desktop Client version

3.8.0

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 3.3.6 to 3.4.0)

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

Are you using an external user-backend?

[X] Default internal user-backend
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Nextcloud Server logs

This seems to be the initial issue (not fully sure):

{"reqId":"zCH6yCfRIeEVQuTq6QP7","level":3,"time":"2023-04-03T20:34:34+02:00","remoteAddr":"xxx.xxx.xxx.xxx","user":"Username","app":"no app in context","method":"DELETE","url":"/ocs/v2.php/apps/end_to_end_encryption/api/v1/lock/871525","message":"Intermediate meta-data file missing","userAgent":"Mozilla/5.0 (Windows) mirall/3.8.0stable-Win64 (build 20230331) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"25.0.5.1","exception":{"Exception":"OCA\\EndToEndEncryption\\Exceptions\\MissingMetaDataException","Message":"Intermediate meta-data file missing","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/end_to_end_encryption/lib/Controller/LockingController.php","line":133,"function":"saveIntermediateFile","class":"OCA\\EndToEndEncryption\\MetaDataStorage","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"unlockFolder","class":"OCA\\EndToEndEncryption\\Controller\\LockingController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/ocs/v1.php","line":63,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/ocs/v2.php","line":23,"args":["/var/www/nextcloud/ocs/v1.php"],"function":"require_once"}],"File":"/var/www/nextcloud/apps/end_to_end_encryption/lib/MetaDataStorage.php","Line":174,"CustomMessage":"--"}}

Nothing in logs for the client messages "Error with the metadata. Getting unexpected metadata format." !!!

Additional info

You should maybe have a look at what you did in e. g. (quick screening for E2E relevant changes)

I downgraded to 3.7.4 and everything is working as expected/normal! No more sync issues or error messages. Just a bit of data loss and loss of 4 hours... thanks.

Currently in rage mode and not willing to upgrade to 3.8.0 again to provide desktop client logs. Sorry.

grosshau commented 10 months ago

I am affected by this too. Any updates planed on this in the near future?

Phrow commented 5 months ago

After spending another 2 hours I am giving up now. Meanwhile desktop client only syncs encrypted files and complains about missing encryption information.

I have the very same issue with the desktop client on windows 10 grafik I'm using desktop client version 3.13.0 and the server is a managed nextcloud offer (v28.0.3, end_to_end_encryption: 1.14.3), so I don't really have raw file access to try some offline decryption. Interestingly I'm also using the iOS app version 5.3.0.21, which is showing me error messages like "Unable to decode the metadata file", but not in the same folders in which I have sync issues on my desktop. So, on my desktop I can still see some files, which are inaccessible on the phone and vice versa - but unfortunately I never get the whole set of all my data together. Any ideas how this can be fixed? I tried a reinstall of the desktop app as well as disabling and re-enabling e2e using the mnemonic, but nothing changed. I'm still seeing the same errors in the very same folders with the very same files.

gravelfreeman commented 5 months ago

I'm having this issue as well. I'm running a 3 weeks brand new Nextcloud install. The Nextcloud desktop client version is 3.13.0 which I believe is the latest version.

Step that lead to this error;

Installed E2EE application in webui
Restarted Windows client so that I can see the encryption option
Setup encryption in Windows client
Created a folder, right click and selected encryption
I copied 5 test video files and they all encrypted successfully
Then I added all my video files (roughly 55 GB)

This is where the client started throwing the errors.

I can't really share my logs because there is so much personal information in them which would take a lot of time to anonymize.

If anyone got a fix to get the sync to complete. Right now, no matter what I'm trying the sync fails and Nextcloud is useless.

MichaelL93 commented 4 months ago

Same for me now. Added a new file to the E2E folder, restarted my mac -> missing metadata error. Re-sync, remove the folder & re-add it, nothing currently resolves it. nextcloud server version 28.0.6. Client 3.13.0. e2e version 1.14.5

mainboarder commented 3 months ago

Now I run into that issue. Server 29.0.4 Client 3.13 E2EE 1.15.2

Am I questioning everything, because of this highly active, but still unresolved thread? - Yes.

Beside this error, the sync seems to work. A file changed on client1, generating this error, is in the expected state on client2.

mainboarder commented 3 months ago

I was able to solve the conflict.

(Enter docker container) Scan for files in the path with problem Restart/Refresh Clients

root@server:/PATCH/TO/NC/data/USER/files/dir1/dir2# docker exec -u 33 -ti a2c3049d9167 /bin/bash
www-data@22ab8669e9db:~/html$ ./occ files:scan       
Please specify the user id to scan, --all to scan for all users or --path=...
www-data@22ab8669e9db:~/html$ ./occ files:scan --path USER/files/dir1/dir2
Starting scan for user 1 out of 1 (USER)
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 6       | 35    | 0   | 1       | 1       | 0      | 00:00:00     |
+---------+-------+-----+---------+---------+--------+--------------+

executed commented 2 months ago

This error affects me too.

occ files:scan didn't help - why would it? :)

Server: 29.0.5. Windows client: 3.13.3

UPD: I think I was able to fix this... We will see how it goes. Ping me to post the solution if you have problems with the PC client and not the app client + you have an unencrypted backup of problematic data + Docker installation (could be a showstopper if not Docker).

nextcloud / desktop