503 Encryption not ready: multikeydecrypt with share key failed

[CamZie]

Steps to reproduce

1. enable encryption
2. upload and download files

Expected behaviour

Nextcloud should allow downloading of files without any errors.

Actual behaviour

Cannot download some files. User is receiving errors that the server is temporarily unavailable (503) or that the server is in maintenance.

Server configuration

Operating system: Debian 8.10

Web server: NGINX 1.12

Database: MariaDB 10.0

PHP version: PHP 5.6

Nextcloud version: 12.0.2

Updated from an older Nextcloud/ownCloud or fresh install: Updated from an older Nextcloud version.

Signing status:

Signing status

``` No errors have been found. ```

List of activated apps:

App list

``` Enabled: - activity: 2.5.2 - admin_audit: 1.2.0 - bookmarks: 0.10.1 - bruteforcesettings: 1.0.3 - calendar: 1.5.7 - comments: 1.2.0 - contacts: 2.0.1 - dav: 1.3.0 - encryption: 1.6.0 - federatedfilesharing: 1.2.0 - files: 1.7.2 - files_pdfviewer: 1.1.1 - files_sharing: 1.4.0 - files_texteditor: 2.4.1 - files_trashbin: 1.2.0 - files_versions: 1.5.0 - files_videoplayer: 1.1.0 - firstrunwizard: 2.1 - gallery: 17.0.0 - logreader: 2.0.0 - lookup_server_connector: 1.0.0 - mail: 0.7.9 - nextcloud_announcements: 1.1 - notes: 2.3.2 - notifications: 2.0.0 - oauth2: 1.0.5 - password_policy: 1.2.2 - provisioning_api: 1.2.0 - qownnotesapi: 17.5.0 - serverinfo: 1.2.0 - sharebymail: 1.2.0 - systemtags: 1.2.0 - tasks: 0.9.5 - theming: 1.3.0 - twofactor_backupcodes: 1.1.1 - updatenotification: 1.2.0 - workflowengine: 1.2.0 Disabled: - federation - files_external - survey_client - user_external - user_ldap ```

Nextcloud configuration:

Config report

``` "system": { "instanceid": "ocpom4ncgfhghkwru", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***REMOVED SENSITIVE VALUE***" ], "datadirectory": "\/mnt\/***REMOVED SENSITIVE VALUE***\/data", "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "12.0.2.0", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "localhost", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "logtimezone": "Europe\/Zurich", "installed": true, "theme": "***REMOVED SENSITIVE VALUE***", "enable_previews": true, "memcache.local": "\\OC\\Memcache\\APCu", "enable_avatars": false, "logdateformat": "Y-m-d_H:i:s", "updatechecker": false, "log_type": "errorlog", "logfile": "", "loglevel": 2, "customclient_desktop": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "trashbin_retention_obligation": "auto,90", "activity_expire_days": 90, "preview_max_scale_factor": 1, "preview_max_filesize_image": 10, "skeletondir": "***REMOVED SENSITIVE VALUE***", "mail_from_address": "no-reply", "mail_smtpmode": "php", "mail_smtpauthtype": "LOGIN", "mail_domain": "***REMOVED SENSITIVE VALUE***"} ```

Are you using encryption: yes

Client configuration

Browser: Operating system: Nextcloud-iOS/2.19.2

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

``` 2018/02/10 04:14:07 [error] 32243#32243: *2115256 FastCGI sent in stderr: "PHP message: [owncloud] [webdav][4] Exception: {"Exception":"Sabre\\DAV\\Exception\\ServiceUnavailable","Message":"Encryption not ready: multikeydecrypt with share key failed:error:0906D06C:PEM routines:PEM_read_bio:no start line","Code":0,"Trace":"#0 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(85): OCA\\DAV\\Connector\\Sabre\\File->get()\n#1 [internal function]: Sabre\\DAV \\CorePlugin->httpGet(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#2 \/var\ /www\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#3 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event \\EventEmitter->emit('method:GET', Array)\n#4 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV \/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#5 \/var\/www\/nextcloud\/apps\/dav\/appinfo\/v1\/webdav.php(71): Sabre\\DAV\\Server->exec()\n#6 \/var\/www\/nextclo" while reading response header from upstream, client: ***REMOVED SENSITIVE VALUE***, server: ***REMOVED SENSITIVE VALUE***, request: "GET /remote.php/webdav/Photos/2018/01/18-01-19%2018-37-42%200433.jpg HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "***REMOVED SENSITIVE VALUE***" ```

[RandieM]

Same problem here with two different Nextcloud 12.0.2 installations. One installation is running on Debian 8 and the other is running on Debian 9, for what it's worth. The rest of my set-up is pretty much the same as @CamZie's. Any ideas?

[tflidd]

@schiessle

[albertogscotti]

Same problem on 13.0.2. Happens on sharing encrypted directories / files. Also: php occ encryption:migrate throws a lot of errors "An unhandled exception has been thrown: ArgumentCountError: Too few arguments to function OCA\Encryption\Migration::__construct()"

[hostingnuggets]

Same problem here too with 13.0.1.

@schiessle what is the status or progress regarding this encryption related bug?

[berho]

I have the same problem on 13.0.2! A lot of files can not be syncronized over dav. This Version of NextCloud is not stable to use in a productive environment!!

How can I get back my files??

[m33m33]

Same problem here on 13.0.4 stable release. Server side encryption activated = impossible to share files. <!> This encryption feature should be disabled on the stable/production releases <!>

[CamZie]

Is there any updates or news for this issue? This is starting to be a big problem since it is impossible to access the files anymore...

[Escubaer]

Is this reproducable with the newest versions eg. 12.0.9? https://github.com/nextcloud/server/wiki/Maintenance-and-Release-Schedule if not it seems rather something for a subscription service - if yes it should get immediate attention, indeed.

[CamZie]

We have upgraded our installation to 13.0.1 and this issue still persists. We haven't been able to identify the cause of this problem.

[RandieM]

@Escubaer, I am having trouble understanding your comment, as several other users (refer to comments above) have already reported that they experience this issue in versions 13.0.1, 13.0.2 and 13.0.4. People keep losing, possibly irreversibly, their data; how does such a major issue qualify as a case for a subscription service?

[Escubaer]

@RandieM I am not working for the vendor, just to make sure. I think I am trying to say and ask if this can be reproduced with a brand new setup eg. with 13.0.4 and with which steps or if this is random and rather happened suddenly in people's running environment. IMHO this will make debugging difficult and therefor it seems maybe more for the subscription/support service. Besides that no developer seems to come up with any idea or solution till now here ...

[Escubaer]

It is also maked as a feature whereas for you guys it sounds like a strong bug ...

[RandieM]

@Escubaer, when it comes to programming, I tend not to believe in "random" events. The described problem is triggered by something, which I am currently unable to identify. This also seems to be the case for @CamZie, according to his/her latest comment.

Besides, you do have a point when you say:

It is also maked as a feature whereas for you guys it sounds like a strong bug ...

I believe that this issue has been assigned the wrong label, as it is certainly not a feature, but a bug /cc @tflidd

[tflidd]

I believe that this issue has been assigned the wrong label, as it is certainly not a feature, but a bug /cc @tflidd

It just says that this topic is related to the server-side-encryption. There are different tags for feature requests ;-)

But regarding the number of users reporting this problem, it is probably more than just a single coincidence. I will put a bug-label to it.

[RandieM]

Thanks @tflidd for the explanation and for the assignment of the new label.

[berho]

I had this wired error once more today and I tested around but can't get any clue why that happens:

Upload from: ------------> Server Thumbnail Creation --------> Download to Windows Client ----------------------------(View and download with Browser)----------------------------------

iOS App ----------------------------> OK --------------------------------> Fail iOS send To NextCloud--------------> OK --------------------------------> Fail Browser (FF on W7) ------------------> OK -------------------------------> Fail Windows Client ----------------------> OK -------------------------------> (uploaded)

Error in Logfile always: Sabre\DAV\Exception\ServiceUnavailable: Encryption not ready: multikeydecrypt with share key failed:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error /htdocs/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 88: OCA\DAV\Connector\Sabre\File->get() [internal function] Sabre\DAV\CorePlugin->httpGet(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))

[CamZie]

Yet another user has this problem and they keep receiving this error when trying to access their files. multikeydecrypt with share key failed:error:0906D06C:PEM routines:PEM_read_bio:no start line

Any news on this as it is getting more and more critical?

[m33m33]

Issue still present on 13.0.5.

As a workaround, is it safe to follow https://docs.nextcloud.com/server/13/admin_manual/configuration_files/encryption_configuration.html and decrypt files with occ ?

No, or I may do something wrong...

After using : php occ encryption:decrypt-all user1

The files are still encrypted on the storage, and users get a "bad signature" on all files. Better have a good backup.

In nextcloud.log : "Exception: {\"Exception\":\"OCP\\Encryption\\Exceptions\\GenericEncryptionException\",\"Message\":\"Bad Signature\",\"Code\":0,\"Trace\":\"#0 \\/mnt\\/sd0d\\/usr\\/pkg\\/share\\/nextcloud\\/apps\\/encryption\\/lib\\/Crypto\\/Crypt.php(465)

A decrypted file "About.txt": file data/user1/files/Documents/About.txt data/user1/files/Documents/About.txt: data ===> should be "text"

First few lines of About.txt: " HBEGIN:oc_encryption_module:OC_DEFAULT_MODULE:cipher:AES-256-CTR:signed:true:HEND---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- " Still encrypted...

[RandieM]

It seems that his behaviour is sometimes triggered by a password change, although I do have users in the same installation that have never changed their password, yet they experience this problem.

Any help would be greatly appreciated, as an increasing number of my users are permanently losing access to their files!

[m33m33]

A clue about this issue: it seems related to public link shared files only:

A. I share a file with a user of my nextcloud instance: the user can open the file. B. I share a file with a public link (url): the link is unusable and throws the multikeydecrypt error message.

[RandieM]

@m33m33, thanks for posting. Initially, I also thought that this was the case, but, in my experience, it does not only happen with shared files.

[CamZie]

Are there any updates or news for this issue?

Just as @RandieM and @m33m33 mentioned, I have also noticed that these are mostly triggered by a password change or shared files, but some of my users also do not have either of them but are still experiencing this problem. Any help would be greatly appreciated.

[m33m33]

Bug still lives in v14. Exterminate. Exterminate. Exterminate.

Considering total NC removal under users grunts.

" Can't read file multikeydecrypt with share key failed:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error "

[albertogscotti]

Dropping NC too

[m33m33]

Another clue about this issue: it seems image format files are not affected.

A. I share a picture (.jpg) with a public link : the destination user can open the link and the image shows in NC viewer. B. I share a document (.pdf, .odt...) with a public link : the link is unusable and throws the multikeydecrypt error message.

[hostingnuggets]

@m33m33 The behavior you describe in your point A might be the effect of the cache: my assumption here is that image files get cached unencrypted and this picture file you shared with a public link is then accessed directly from the cache, that's why it works.

Have a look at my comment here and the answers below on the nextcloud forum: https://help.nextcloud.com/t/nextcloud-14-focus-on-security-and-compliance/36116/2

In my comment I have asked the nextcloud core team why they don't seem to care about fixing and even replying to all the server-side encryption issues...

[m33m33]

@m33m33 The behavior you describe in your point A might be the effect of the cache: my assumption here is that image files get cached unencrypted and this picture file you shared with a public link is then accessed directly from the cache, that's why it works.

You are right. I am fooled by the preview from cache, if I click on "download" the picture don't show and the multikey failure message appears :(

[schiessle]

I have the feeling that this issue mixes many potential different problems together. E.g. the original issue says that the user gets a "503 Nextcloud unavailable or in maintenance mode" which I never saw and I don't know how this could be triggered by the server side encryption. The other error messages posted here make more sense but I still struggle to find the necessary information and what all this reports have in common in order to try to reproduce it.

So my request to everyone in this issue. Can someone of you describe a step by step scenario with the latest Nextcloud version (13.0.6 or 14, because they contain some changes to make the file cache updates more robust) where they can reliable reproduce the issue?

If I have something like this I'm happy to give it another try and see if I can reproduce it.

[m33m33]

NC 14, current user with server side encryption enabled.

1. Upload a new file : picture.jpg

2. Create a public link

3. Give public link to an external anonymous user : the preview is OK. Click on "Download" shows "Can't read file multikeydecrypt with share key failed:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error"

[schiessle]

@m33m33 I just tried it and couldn't reproduce it. Does the error happens for you reliable with all files? Is it a fresh installation or a update from a older version. On which version did you enabled encryption? Do you still use per-user keys or the new master key?

[m33m33]

@schiessle it happens on 100% files. It's an update, I've been updating to all new versions since 13.whatever. I enabled server side encryption with per-user key on v13.whatever, sorry I don't remember the exact release, but it should be the current release as of 16 Jun (when I joined this ticket).

[schiessle]

@m33m33 thanks. I tested it with the master key. I will try it with per user keys as well.

[schiessle]

Sadly I also couldn't reproduce it with per user keys. Can you check this folder data/<owner>/files_encryption/keys/files/<filename>/OC_DEFAULT_MODULE and tell me what's in it?

After you created a public link there should be three files in it:

• <owner>.shareKey
• fileKey
• pubShare_<some-random-chars>.shareKey

Also please post the error message you see in the nextcloud.log directly after you tried to download the file from the public link.

[m33m33]

Hi, issue still present in 14.0.1.

Yes, a public link does create three files in OC_DEFAULTMODULE: fileKey {user}.shareKey pubShare{8 caracters}.shareKey

Log extract directly after a click on "download:" https://paste.ee/p/NG6R2

[m33m33]

Another clue about this issue to help you narrow the search :

1. Internal user create a folder and share it with public link.
2. Foreign user upload files to the folder
3. Both internal and foreign user can preview and download files without encryption error.

[chrsch]

This is still an issue in 14.0.8, is this solved in NC15/16 already?

[m33m33]

Still in 15.x

[wandoliver]

Just ran into the same problem on NC 15. No previews are generated anymore, always running into this error message. For both, shared and non-shared images.

[chrsch]

@schiessle I would like to nail this down in current NC15. Do you have some hints where to start digging in the encryption app? Thanks.

[m33m33]

@schiessle I would like to nail this down in current NC15. Do you have some hints where to start digging in the encryption app? Thanks.

Sorry, had to take the NC service down because of data loss with this issue. Can't provide much more help now :/

It's that simple: setup a NC, use it a little, then enable server side encryption, use it a little more and it may happen.

[moonmeister]

Issue still happening in v16...in fact it only started happening for me after i upgraded to v16.

[luisBSBDATA]

Also happening here on 13.0.2.. It was running just fine for more than a year now and suddenly today it started to happen. I'm extremely scared as it's sensitive and important data and can't seem to recover it

[hurrrrricaaaaane]

Same here with nextcloud 16 after enabling 2 Factor Auth and E2E encryption

[Zmmfly]

Same problem after decrypt all files, but still encrypted.

[Zmmfly]

Version:16.3

[bjo81]

Same sh*t, different instance on 16.0.3.

[ghost]

The same here: https://github.com/nextcloud/notes/issues/328

Nextcloud: 16.0.4

Only that in this case it is not about shared files, but about notes of the Notes android app.

The whole thing started with the activation of 2FA. Since then the files of the notes (via android app and web browser) have been duplicated when trying to open them and the name has been extended by "(2)". So "filename/note.txt" to "filename/note (2).txt".

This worked for a while and now (probably after an update) it doesn't let any of the files/notes open anymore, but it always comes the already known error message.

Can it be that Nextcloud has stored or finds two different keys and does not know which one to take? Can I influence this somehow?

I have deactivated 2FA again, but the problem remains. Currently "Encrypt the home storage" is still activated. Would it change anything if I deactivated it or is all data useless in any case?

[ghost]

Even after deactivating the server-side encryption, it is not possible to access the data. The data remains encrypted with an (now) unknown key.

Maybe already encrypted data was encrypted again with another key. The first key does not exist anymore and therefore decryption is impossible.

I think that an update has destroyed everything and there is no possibility to access the data anymore.

This is a real worst-case scenario for a software that is widely used by the ITZBund (Informationstechnikzentrum Bund in Germany), as well as by authorities in France, Sweden and the Netherlands, and really shameful.

[Fjodor42]

Also bitten by this, and will be happy to provide any logs deemed necessary to nail this.

@schiessle: Are you still involved in this?

[Fjodor42]

To clarify:

I am not entirely unwilling to actually grant access to my instance, or to the actual server itself, provided that the developer to be such such access can be vouched for by other (lead) developers.

What I am seeing specifically, is the following, on a file in a folder that is both shared with me from a user, and has a share link, with the file opening fine for my own user, but throwing an error when accessed through the share link to the folder (the listing of the folder seems fine in both cases):

Can't read file

multikeydecrypt with share key failed:error:04065084:rsa routines:rsa_ossl_private_decrypt:data too large for modulus

@schiessle , if you're there, or anyone else if not: Let me know if there is anything I can do to help debug

Googling "data too large for modulus" doesn't really get me anywhere, except that it could be related to padding, could be related to the RSA implementation as it pertains to async features, and could be something else entirely, but my Google-fu might be weak here...