"Server E2EE version 1.2, not compatible" plus "serious internal error in end-to-end encryption"

bcutter commented 9 months ago

Steps to reproduce

See steps at https://github.com/nextcloud/desktop/issues/5918#issuecomment-1962958510
Create a new folder in an E2EE folder (using the Windows client)
Open NC iOS app and open a E2EE folder

Expected behaviour

Folder content is shown, including the plain text name of the new created folder.

Actual behaviour

Error message is shown. Newly created folder is not shown in plain text.

Screenshots

Accessing E2EE folder with a newly created sub folder: grafik

Going to an older subfolder which contains another newly created sub folder: grafik

Status of E2EE on iOS: grafik

Logs

When only accessing E2EE folder initially there was once in the server log:

[no app in context] Fehler: OCA\EndToEndEncryption\Exceptions\MissingMetaDataException: Intermediate meta-data file missing at <<closure>>

0. /var/www/nextcloud/apps/end_to_end_encryption/lib/Controller/LockingController.php line 158
   OCA\EndToEndEncryption\MetaDataStorage->saveIntermediateFile()
1. /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 230
   OCA\EndToEndEncryption\Controller\LockingController->unlockFolder()
2. /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 137
   OC\AppFramework\Http\Dispatcher->executeController()
4. /var/www/nextcloud/lib/private/AppFramework/App.php line 183
   OC\AppFramework\Http\Dispatcher->dispatch()
5. /var/www/nextcloud/lib/private/Route/Router.php line 315
   OC\AppFramework\App::main()
6. /var/www/nextcloud/ocs/v1.php line 65
   OC\Route\Router->match()
7. /var/www/nextcloud/ocs/v2.php line 23
   require_once("/var/www/nextcloud/ocs/v1.php")

DELETE /ocs/v2.php/apps/end_to_end_encryption/api/v1/lock/1038380?e2e-token=r6rXXXXYevR5h8yeXXXXVG2YlrVXXXXx24xPttVXXXXbDph8UXXXXuXuMyXXXXcu
from xxx.xxx.xxx.xxx by Username at 2024-02-25T15:27:41+01:00

(few parts masked with xxx | XXX)

Now when creating a new E2EE folder on the iOS app, after it synced to a Windows endpoint, trying to delete that folder on the Windows endpoint, desktop client shows an error and server log contains:

[webdav] Fehler: OCA\DAV\Connector\Sabre\Exception\Forbidden: Write access to end-to-end encrypted folder requires token - no token sent at <<closure>>

0. /var/www/nextcloud/apps/end_to_end_encryption/lib/Connector/Sabre/LockPlugin.php line 143
   OCA\EndToEndEncryption\Connector\Sabre\LockPlugin->verifyTokenOnWriteAccess()
1. /var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
   OCA\EndToEndEncryption\Connector\Sabre\LockPlugin->checkLock()
2. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 456
   Sabre\DAV\Server->emit()
3. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 253
   Sabre\DAV\Server->invokeMethod()
4. /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 321
   Sabre\DAV\Server->start()
5. /var/www/nextcloud/apps/dav/lib/Server.php line 365
   Sabre\DAV\Server->exec()
6. /var/www/nextcloud/apps/dav/appinfo/v2/remote.php line 35
   OCA\DAV\Server->exec()
8. /var/www/nextcloud/remote.php line 172
   require_once("/var/www/nextcl ... p")

DELETE /remote.php/dav/files/Username/Test
from XXX.XXX.XXX.XXX by Username at 2024-02-25T15:52:06+01:00

If now content is added to a (with iOS app) newly created E2EE folder, it will never be synced to a Windows endpoint (trying forever) - nothing in server logs for this: grafik

grafik

Reasoning or why should it be changed/implemented?

There's obviously a serious E2EE issue - again! So move this to whereever you think: iOS, desktop, end_to_end_encryption.

I think it all started with https://github.com/nextcloud/desktop/issues/5564 back then - BUT I had a rather working setup (except for https://github.com/nextcloud/desktop/issues/5918 on Windows endpoints/the desktop client).

Environment data

iOS version: 16.7.5

Nextcloud iOS app version: 5.1.0.7

Server operating system: Raspberry Pi OS

Web server: nginx

Database: MariaDB

PHP version: 8.3

Nextcloud version: 27.1.5

Seriously: how can I start with E2EE from scratch? I only and every see issues with it, despite the fact I need it.

bcutter commented 9 months ago

Update: I went through https://github.com/nextcloud/end_to_end_encryption/issues/32#issuecomment-466037407 and reset all E2EE keys and meta data to basically start from scratch.

After freshly creating a new folder and setting it as E2Eencrypted on a Windows endpoint, I accessed it using the iOS client. After providing the new passphrase, I again get this ⚠️⚠️⚠️

grafik

Interestingly:

Create a new E2EE folder on iOS client, upload data
Data is synced to and accessible on Windows endpoint
BUT: once the Windows endpoint adds data to that folder, again NOT READABLE on iOS (and the red error is shown)

To sum up:

Everything coming from the iOS 5.1.0.7 client can be read on other E2EE clients (Windows desktop client 3.12.0)
Everything coming from Windows desktop clients 3.12.0 can not be read on the iOS 5.1.0.7 client

So, where's the core issue here? And:

So why does the app reference "End-to-End Encryption 1.2" at all? On the server v1.13.1 is installed. ➡️ Has this maybe been introduced with latest iOS app 5.1.0.7 https://github.com/nextcloud/ios/releases/tag/5.1.0 release (I installed 4 days ago)?

For now my E2EE is completely broken / unusable on iOS.

bcutter commented 9 months ago

Another update: tested the E2EE sync with

several other Windows endpoints. Works flawlessly in both directions.
an Android client. Works flawlessly in both directions.

SO: E2EE is working fine for ALL synced clients, except iOS. Everything is pointing back to the iOS app ⚠️

This PLUS the fact I did not use E2EE content in the iOS app for a few days PLUS the fact nothing changed on the server side really makes me guess https://github.com/nextcloud/ios/releases/tag/5.1.0 broke something here.

eibex commented 9 months ago

This could also be due to the Windows client using E2EEv2 for new folders and migrating older ones since recent versions.

bcutter commented 9 months ago

This could also be due to the Windows client using E2EEv2 for new folders and migrating older ones since recent versions.

I don't know if this is the case (please note: last Windows desktop client update to 3.12 was at something like 18/19th of February, many days before the issue came up). I also don't know why it works for Android. I only see the NC iOS app not working at all now when it comes to E2EE.

I provided everything I can, now someone with knowledge of the app and E2EE needs to look at that. Please. Thank you.

marinofaggiana commented 9 months ago

don't worry soon a fix

bcutter commented 9 months ago

don't worry soon a fix

I really like that post - even every single word is finest English, I can almost feel the Mario-alike Italian groove in it (checked your profile and indeed - Italian!) - love it 🙂

Back on topic:

Looking forward to that fix - and hopefully a user understandable explanation what actually went wrong here, as I still don't understand it.
Please note I don't have a test flight so I won't be able to test the fix prior it is actually shipped as part of a new release in the app store.

bcutter commented 9 months ago

don't worry soon a fix

Any estimation (definition of "soon")?

Bothers me on an everyday basis...

marinofaggiana commented 8 months ago

Please use the version in TestFlight !

bcutter commented 8 months ago

I can't.

Please note I don't have a test flight so I won't be able to test the fix prior it is actually shipped as part of a new release in the app store.

https://github.com/nextcloud/ios/issues/2809#issuecomment-1967532630

marinofaggiana commented 8 months ago

it's in readme, it's sufficient read it

https://testflight.apple.com/join/RXEJbWj9

bcutter commented 8 months ago

What does the way back from beta/TestFlight release to stable look like?

Edit: had a look at TestFlight. Oh wow, all my usage information is sent to Nextcloud and Apple. Ehm, no. Now I remember why I never used TestFlight.

bcutter commented 8 months ago

@marinofaggiana please note the update (v5.2.0.9, installed from the app store, app also force closed etc. to prevent any caching issues) did unfortunately not fix the issue. It remains unchanged:

marinofaggiana commented 8 months ago

Hi @bcutter your error happen when the metadata is illegible, what was it created or modified with? I think not with iOS because I have make test of V 1.2 and works.

beposec commented 8 months ago

I was able to reproduce it. A directory created and encrypted on the desktop app 3.12 can not be accessed on iOS. A directory created and encrypted on iOS can be accessed on desktop. So maybe there is something wrong with the Desktop App?

But i was not able to choose the E2E directory to Auto Upload Photos. Is this intended behaviour?

bcutter commented 8 months ago

Created on latest 3.12/3.12.1 desktop client (Windows).

Side information: For me as a user it doesn't matter which client creates, accesses, edits or deletes E2EE content. The server component and all E2EE clients need to take care they are compatible. That's what we expect especially when running up to date versions.

How to proceed? Do you @marinofaggiana want to give the desktop client or server component experts a push?

marinofaggiana commented 8 months ago

I was able to reproduce it. A directory created and encrypted on the desktop app 3.12 can not be accessed on iOS. A directory created and encrypted on iOS can be accessed on desktop. So maybe there is something wrong with the Desktop App?

Will make test with our Desktop team

But i was not able to choose the E2E directory to Auto Upload Photos. Is this intended behaviour?

Yes, Encryption cannot be performed in the background, so autoloading was deliberately excluded.

beposec commented 8 months ago

Yes, Encryption cannot be performed in the background, so autoloading was deliberately excluded.

Thanks for that Info. So sad! On Android this seem to work so i was hoping its a bug on iOS. Then it should maybe not allowed to encrypt the chosen Auto Upload directory. But sorry for hijacking this issue.

bcutter commented 8 months ago

Will make test with our Desktop team

Thanks. With 5.2.1 E2EE still broken.

marinofaggiana commented 8 months ago

Soon a desktop update.

bcutter commented 8 months ago

Soon a desktop update.

@marinofaggiana While I could see some E2EE related fixes in https://github.com/nextcloud/desktop/releases/tag/v3.12.2 like

I could not spot any difference (and updating to 3.12.2 made absolutely zero difference on the iOS app). I even can't judge if those changes affect this issue here at all in a positive way. Therefore: can you please link an issue or even PR of the desktop repo here so we can watch the actual progress? Thank you.

marinofaggiana commented 8 months ago

it's fixed @bcutter try it

bcutter commented 8 months ago

More details please. As I wrote:

and updating to 3.12.2 made absolutely zero difference on the iOS app

E. g., is there a need to re-create an E2EE folder using the desktop client to resolve the conflicts on the iOS side?

Your information is very minimalistic :-)

marinofaggiana commented 8 months ago

No, was only an error of decode metadata version, so nothing happened to the data. #https://github.com/nextcloud/desktop/pull/6543

bcutter commented 8 months ago

OK. Here is what I did: 1) Updated desktop client on one of several Windows endpoints to 3.12.2 2) Checked back to the iOS app

Expectation: Issue is solved Reality: Nothing changed. Still error message.

311287301-3161e94d-e4dd-4048-9c90-a669c0c10cbc

So please assist @marinofaggiana.

eibex commented 8 months ago

For more recent folders, the app also says "Server E2EE version 2.0, not compatible".

It looks like the issue is with both legacy and modern versions of E2EE.

bcutter commented 8 months ago

So it's (still) a thing on the iOS side? Or even the server (E2EE app)? For sure the changes on the desktop endpoint effectively changed nothing. Unfortunately...

marinofaggiana commented 8 months ago

Hi, calm :D

1 - please report version NC iOS, Server, and Desktop 2 - it's a old e2ee folder or a new folder 3- have you create a new e2ee folder ?

bcutter commented 8 months ago

Hi, calm :D

Not possible. Speed is key here.

1 - please report version NC iOS, Server, and Desktop

iOS: 5.2.1.0
Server: 27.1.7
Desktop: 3.12.2

2 - it's a old e2ee folder or a new folder

For me: "old" E2EE folder (created with desktop client < 3.12.2)
For @eibex I think: "new"

3- have you create a new e2ee folder ?

See 2.

Edit: Same with latest iOS version. New look (icons changed, didn't they?), same behavior.

marinofaggiana commented 8 months ago

@bcutter can you create a new folder e2ee and try it ? (desktop <> iOS) thanks

bcutter commented 8 months ago

@bcutter can you create a new folder e2ee and try it ? (desktop <> iOS) thanks

@marinofaggiana Yes I can - and I did. Results:

1) Readable on iOS (also writable) without an error message

2) Strangely, a text file with content "Test encrypted" and a carriage return is shown as this:

Therefore:

a) Will there be a migration on the desktop client side from the "old" to the "new" E2EE version (I suspect 1.2 to 2.0)? If not how would you handle that for all the users?
b) What about number 2. Is it an error on the desktop client side, the server part or a "preview" issue on the iOS app side? Hopefully not an integrity issue...

Even working around a) manually because of b) I don't trust the whole thing enough yet to migrate my E2EE content manually. Please advise.

Edit/Update:

Once I (temporarily) renamed the existing E2EE folder (like E2EE-encrypted --> E2EE-encrypted-renamed), it was immediately readable by the iOS app. So it seems like the root folder needs to be changed by the desktop client to upgrade from 1.2 to 2.0, right? That might hopefully be a relevant information to you in terms of migration path.

marinofaggiana commented 8 months ago

@bcutter in reality the thing is very simple, the desktop version introduced a version error in the metadata file, the iOS client was therefore unable to read it generating an error, just re-modify (a rename or a delete or add a new file) the desktop part with the correct version (3.12.2) and everything is resolved. we always talk about version 1.2, with 2.0 this doesn't happen. Regarding file encryption, nothing changes from version 1.2 to 2.0 so if there are ever any problems the encrypted file is safe.

About: 316249173-a7ca6093-7100-4124-81fd-91b798cdc328 PNG

I alerted the desktop team to check the end of file

eibex commented 8 months ago

with 2.0 this doesn't happen

I am seeing the same error with folders using version 2.0 (desktop, ios, server, and e2ee at latest versions)

bcutter commented 8 months ago

Please note: E2EE was working fine the last days. Today I updated from NC 27.1.8.1 to NC 28.0.4.1, E2EE app got updated from 1.13.1 to 1.14.5.

Once I tried to add a file in a e2ee folder on a Windows endpoint, I got this:

NC 28_E2EE_iOS

NC 28_E2EE_Windows

No sync possible for the whole affected encrypted e2ee anymore.

I tested this:

newly created folder (on Windows client): no issues
newly created folder (on iOS client): no issues

Therefore I had to disable e2ee app, switch to the e2ee folder on the web interface and deleted it, than re-enabled e2ee app. Now I could create a new e2ee folder with the same name.

Whatever you guys fixed (thanks for this), it seems to not survive a NC (major) update, in my case from v27 to v28 series.

allexzander commented 8 months ago

This could be that problem when the server returns 500 on GET metadata in case the metadata was first uploaded via the v1 endpoint and then was being fetched (GET) via the v2

bcutter commented 8 months ago

This could be that problem when the server returns 500 on GET metadata in case the metadata was first uploaded via the v1 endpoint and then was being fetched (GET) via the v2

Good point. Now I re-created that folder again so it is a v2 e2ee folder. BUT it should have been migrated already (https://github.com/nextcloud/ios/issues/2809#issuecomment-2016516189). Everything was working until the NC server and e2ee app update.

eibex commented 8 months ago

It was never working for me with v2 folders either and I've always been on v28 since the issue started.

Also the desktop client isn't syncing anymore showing metadata setup errors as linked in https://github.com/nextcloud/desktop/issues/6452#issuecomment-2023975679_

eibex commented 7 months ago

E2EE has been unusable for over a month now, with the desktop client not being able to sync both e2ee and regular files.

bcutter commented 7 months ago

For me everything is working fine for some while. Even I fear saying that as purely saying it might provoke something to break again.

gravelfreeman commented 6 months ago

I'm seeing a lot of people mentioning 2 things.

Bug might due to iOS app
Bug might be because it's syncing old metadata folder

I'm having this issue as well but I'm not using iOS nor old metadata. I'm running a 3 weeks brand new Nextcloud install. The Nextcloud desktop client version is 3.13.0 which I believe is the latest version.

Step that lead to this error;

Installed E2EE application in webui
Restarted Windows client so that I can see the encryption option
Setup encryption in Windows client
Created a folder, right click and selected encryption
I copied 5 test video files and they all encrypted successfully
Then I added all my video files (roughly 55 GB)

This is where the client started throwing the errors.

I can't really share my logs because there is so much personal information in them which would take a lot of time to anonymize.

nextcloud / ios