search

owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.36k stars 2.06k forks source link

Can't migrate encryption keys from cli #16928

Closed redhell closed 9 years ago

redhell commented 9 years ago

Hi, on my test installation i installed the newest master revision and activated the server side encryption module. (oC 8.1 latest master, php 5.6.10, mariaDB 10.0) In my logs this warning rises: Warning no app in context Installation is in transit between the old Encryption (ownCloud <= 8.0) and the new encryption. Please enable the "Default encryption module" and run 'occ encryption:migrate'

I followed the warning: sudo -u USER /path/to/occ encryption:migrate

The cli got following output: Reorganize system folder structure PHP Fatal error: Call to a member function getRelativePath() on a non-object in /path/to/apps/files_trashbin/lib/storage.php on line 123

karlitschek commented 9 years ago

@schiesbn What do you think?

schiessle commented 9 years ago

any additional errors in your owncloud.log and/or apache error log. Would be great if you could use the issue template to provide some more informations https://raw.githubusercontent.com/owncloud/core/master/issue_template.md Thanks!

redhell commented 9 years ago

specs: See #14139 (just 8.1)

As i tried it today again: (path shortened) opendir(/data/files_encryption/keys): failed to open dir: No such file or directory at /lib/private/files/storage/local.php#105

followed by the error from above. nginx/apache2 looks normal.

redhell commented 9 years ago

{"reqId":"T4qWGN\/EV6aFvB0lTyNB","remoteAddr":"","app":"PHP","message":"opendir(\/var\/www\/*\/data\/filesencryption\/keys): failed to open dir: No such file or directory at /lib\/private\/files\/storage\/local.php#106","level":3,"time":"2015-07-07T14:11:53+02:00"} {"reqId":"T4qWGN\/EV6aFvB0lTyNB","remoteAddr":"","app":"PHP","message":"Call to a member function getRelativePath() on a non-object at _/apps\/files_trashbin\/lib\/storage.php#123","level":3,"time":"2015-07-07T14:11:53+02:00"}

Still can't migrate my keys on stable8.1! @karlitschek @schiesbn

The directory keys does not exist in /data/files_encryption

jolorant commented 9 years ago

Same here.

  1. Have a working 8.0.4 owncloud installation
  2. Install owncloud 8.1 from CentOS 6 repo
  3. Be advised to enable default encryption module and do so
  4. cd /path/to/owncloud/server && sudo -u apache ./occ encryption:migrate
  5. Output: Reorganize system folder structure PHP Fatal error: Call to a member function getRelativePath() on null in /var/www/cloud.atrics.de/apps/files_trashbin/lib/storage.php on line 123

What happened: {"reqId":"r+sBtMCH7\/3zLGxfQ1kT","remoteAddr":"","app":"PHP","message":"opendir(\/path\/to\/owncloud\/data\/files_encryption\/keys): failed to open dir: No such file or directory at \/path\/to\/owncloud\/www\/lib\/private\/files\/stor age\/local.php#106","level":3,"time":"2015-07-07T13:14:48+00:00"}

What I expected to happen: successful conversion of encryption keys to new (8.1) format

OS: CentOS 6.6 Web: apache httpd 2.2.15 DB: MySQL 5.1.73 PHP: php56u-5.6.10-1.uis.centos6.x86_64 OC: 8.1 now, upgraded from 8.0.4 (8.0.5 vanished from the repo right before my eyes)

Enabled apps: Enabled:

Internal storage, encryption used, no external user-backends, no ldap etc.

jknockaert commented 9 years ago

Same error here migrating from 8.0.5 to 8.1.

jolorant commented 9 years ago

After logging out and in again, files belonging to admin are now inaccessible. I tried link-sharing one, but when accessing such a file, I get a 500 server error. The error_log says: PHP Fatal error: Call to a member function isReadable() on null in /var/www/cloud.atrics.de/lib/private/files/storage/wrapper/encryption.php on line 288

schiessle commented 9 years ago

@jknockaert any chance that you can debug this? I can't re-produce it.

jknockaert commented 9 years ago

@schiesbn I'm working on it, but the upgrade to 8.1 is generally rougher than expected.

schiessle commented 9 years ago

Thanks a lot @jknockaert

Now I tried it again and can reproduce it. The bug is in the trashbin and not in encryption. Disabling the "deleted files" app and re-run the occ command should work. After the migration was completed "deleted files" can be enabled again.

jknockaert commented 9 years ago

@schiesbn After disabling the trashbin app I get a different error Reorganize system folder structure

[Doctrine\DBAL\Exception\UniqueConstraintViolationException]
An exception occurred while executing 'UPDATE "oc_appconfig" SET "appid" = ? WHERE "appid" = ?' with params ["encryption", "files_encryption"]:
SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "oc_appconfig_pkey"
DETAIL: Key (appid, configkey)=(encryption, installed_version) already exists.

[Doctrine\DBAL\Driver\PDOException]
SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "oc_appconfig_pkey"
DETAIL: Key (appid, configkey)=(encryption, installed_version) already exists.

[PDOException]
SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "oc_appconfig_pkey"
DETAIL: Key (appid, configkey)=(encryption, installed_version) already exists.

encryption:migrate [user_id1] ... [user_idN]

jknockaert commented 9 years ago

Further checking the oc_appconfig table I notice that there are entries for both files_encryption and encryption (app_id field). Could it be that enabling the new encryption app created new entries in that table, whereas these should actually be created by the occ script upon migration (with the app being enabled only after that)?

schiessle commented 9 years ago

I'm at the same point now. But I think that only happens because we are in a intermediate state now. Everytime we run the migrate script a pubShareKey gets created and the ID gets written to the database, then renaming of the old one to the new one fails. I'm quite sure that this doesn't happen if the first update runs successfully. Probably we tried to be to clever in updateDB(). Instead we should have just read all the files_encryption settings, and then insert them one by one to the table and update existing values if they already exists for some reasons but skip the "installed_version" config key. After that we should remove all the files_encryption settings (again beside the "installed_version" key, this one should be only removed in the constructor if everything was finished).

jknockaert commented 9 years ago

@schiesbn Can't confirm your reasoning but it sounds reasonable. Now how do I get out of this intermediate state asap?

schiessle commented 9 years ago

In my case the problem is that we want to rename "files_encryption"->"publicShareKeyId" to "encryption"->"publicShareKeyId" but this fails because "encryption"->"publicShareKeyId" already exists. So the fix for updateDB() is probably not to rename the "appid" from "files_encryption" to "encryption" because this can lead to the UniqueConstraintViolationException but read the files_encryption values, check if the same configkey for appid "encryption" already exists, if yes we update it if not we insert it. This way we avoid the UniqueConstraintViolationException and make sure that we migrate all settings from "files_encryption" to "encryption"

schiessle commented 9 years ago

as a really quick fix you could also do this manually on your database

jknockaert commented 9 years ago

I was thinking of fixing this manually (i.e. updating the publicsharekeyid field for encription with the value from files_encryption), but I expect the occ script still to fail because it will continue to try to update the legacy entries resulting in double keys. Note that the same issues happen for the recoveryKeyId and recoveryAdminEnabled fields if you have them set.

schiessle commented 9 years ago

if you fix it manually and remove the old 'files_encryption" entries afterwards so that nothing is left to rename the script should finish successfully.

jknockaert commented 9 years ago

OK, but then I could as well just remove the entries publicsharekeyid, recoveryKeyId and recoveryAdminEnabled for 'encryption' before running the occ script (which will update the 'files_encryption' ones to 'encryption').

jknockaert commented 9 years ago

And now it says: [InvalidArgumentException]
Command "encryption:migrate" is not defined.
Did you mean one of these?
encryption:status
encryption:enable
encryption:disable
encryption:list-modules
encryption:set-default-module

schiessle commented 9 years ago

OK, but then I could as well just remove the entries publicsharekeyid, recoveryKeyId and recoveryAdminEnabled for 'encryption' before running the occ script (which will update the 'files_encryption' ones to 'encryption').

Yes but they will be re-created before you reach the point where we rename them. At least that was my experience.

schiessle commented 9 years ago

Your error message probably happens because you deleted all "encryption" app settings, also the setting that the encryption module is enabled. So right now the app is again disabled on your system -> no command line tool

jknockaert commented 9 years ago

Sorry, that last message effectively happened because I manually disabled the app.

Peac commented 9 years ago

I've got this one, after "sudo -u www-data php occ encryption:migrate". Seems to be similar.

Reorganize system folder structure

[Doctrine\DBAL\Exception\UniqueConstraintViolationException] An exception occurred while executing 'UPDATE oc_appconfig SET appid = ? WHERE appid = ?' with params [" encryption", "files_encryption"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'encryption-publicShareKeyId' for key 'PRIMARY'

[Doctrine\DBAL\Driver\PDOException] SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'encryption-publicShareKeyId' for key 'PRIMARY'

[PDOException] SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'encryption-publicShareKeyId' for key 'PRIMARY'

encryption:migrate [user_id1] ... [user_idN]

How do I get this back working? I'm not very familiar with MySQL.

jknockaert commented 9 years ago

@schiesbn So I ran delete from oc_appconfig where appid='encryption' and configkey in ('recoveryKeyId','publicShareKeyId','recoveryAdminEnabled'); and immediately after that update oc_appconfig set appid='encryption' where appid='files_encryption';. That allowed me to run the occ migration script without errors. However, all files are now unreadable. Any ideas? Nothing in the owncloud log save for errors caused by the unreadability (e.g. from the process creating thumbnails).

jknockaert commented 9 years ago

I tried to move the files_encryption folders back from a 8.0.5 backup prior to upgrade and run the migration script again. Some files are now readable, others are not. Very messy.

jknockaert commented 9 years ago

Sorry hit the wrong button.

ghost commented 9 years ago

Linux, Debian php 5.6 Apache 2.4.10 Mysql 5.5.43 Owncloud from 8.0.4 to 8.1.0

user@cloud:/var/www/cloud# sudo -u www-data php occ encryption:migrate Reorganize system folder structure [Doctrine\DBAL\Exception\UniqueConstraintViolationException]
An exception occurred while executing 'UPDATE oc_appconfig SET appid = ? WHERE appid = ?' with params ["encryption", "files_encryption"]:
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'encryption-publicShareKeyId' for key 'PRIMARY'

[Doctrine\DBAL\Driver\PDOException]
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'encryption-publicShareKeyId' for key 'PRIMARY'

[PDOException]
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'encryption-publicShareKeyId' for key 'PRIMARY'

encryption:migrate [user_id1] ... [user_idN]

user@cloud:/data/cloud# cat owncloud.log

{"reqId":"rDcWmB3D0opU7fKVGrue","remoteAddr":"","app":"PHP","message":"opendir(\/data\/cloud\/files_encryption\/keys): failed to open dir: No such file or directory at \/var\/www\/cloud\/lib\/private\/files\/storage\/local.php#106","level":3,"time":"2015-07-08T04:35:15+00:00"}

Same here.

Note: i had to run the occ upgrade a couple of times before i finaly got it running maybe that was the point where files/table entries were created before they should have been created by the encryption migration process?

I still dont understand how to fix this?

phjr commented 9 years ago

Same problem here, first time I ran encryption:migrate I got the "Call to member function", from then on I'm getting the SQL integrity constraint violations.

pgrzywacz commented 9 years ago

I'm actually quite angry right now. Why knowing about this issue was 8.1 even released? There was no warning on the release page: https://owncloud.org/blog/owncloud-8-1-raising-the-bar-on-security-and-performance/

UH-Nerion commented 9 years ago

No news how to fix problem with SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'encryption-publicShareKeyId' for key 'PRIMARY'?

redhell commented 9 years ago

After disabling the Trashbin and getting the [Doctrine\DBAL\Exception\UniqueConstraintViolationException] I deleted the files_encryption rows from the app_config table as @schiesbn mentioned. Rerunning the occ script migrated the keys but all files are not accessible! New files are accessible...

schiessle commented 9 years ago

@jknockaert can you check the keys for the files which are not accessible? Are the keys on the right location?

Please also see https://github.com/owncloud/core/pull/17473 for the database issue

schiessle commented 9 years ago

@redhell can you provide more information. Any message on the web interface if you try to download a file? Anything in your apache error log or your owncloud.log?

schiessle commented 9 years ago

I can no longer re-produce the trashbin issue. :confused: Just did a upgrade with the db fix and with trashbin enabled and everything worked just fine...

jtoloe commented 9 years ago

@schiesbn, @jknockaert I have the same issue and did the manual change to the database. After that the migration worked but I can not access any old files. New files (text files) created through the web client are encrypted and accessible but does not sync using the desktop client. The desktop client gives the following error:

server replied: Service Unavailable (Encryption not ready: multikeydecrypt with share key failed:error:0607A082:digital envelope routines:EVP_CIPHER_CTX_set_key_length:invalid key length)

which looks like an openssl error?

Anyway, the keys are migrated to the right location but the old directories were not removed. I tried remvoving them but the issue persists.

redhell commented 9 years ago

Ok: PDF.js Version 1.0.712 (build: 6969ed4) Nachricht: Unexpected server response (500) while retrieving PDF "https://*/index.php/apps/files/ajax/download.php?dir=%2F&files=ownCloudUserManual.pdf".

and in the error.log

modfcgid: stderr: PHP Fatal error: Call to a member function isReadable() on null in /var/www//lib/private/files/storage/wrapper/encryption.php on line 288, referer: http://_/index.php/apps/files/

ownCloud Log is empty

twouters commented 9 years ago

Here's how I "solved" the issue after disabling the trash bin app.

mysql> select * from oc_appconfig where appid like '%encryption%';
+------------------+----------------------+----------------------+
| appid            | configkey            | configvalue          |
+------------------+----------------------+----------------------+
| encryption       | enabled              | yes                  |
| encryption       | installed_version    | 1.0.0                |
| encryption       | publicShareKeyId     | pubShare_eaa*****    |
| encryption       | recoveryKeyId        | recoveryKey_eaa***** |
| encryption       | types                | filesystem           |
| files_encryption | publicShareKeyId     | pubShare_a8d*****    |
| files_encryption | recoveryAdminEnabled | 1                    |
| files_encryption | recoveryKeyId        | recovery_b3c*****    |
+------------------+----------------------+----------------------+

First thing I noticed was a mismatch of the encryption and files_encryption recoveryKey and pubShare values. I figured the "old" files_encryption values would be the correct ones so I updated the values where appid=encryption

mysql> update oc_appconfig set configvalue='pubShare_a8d*****' where appid='encryption' and configkey='publicShareKeyId';
mysql> update oc_appconfig set configvalue='recovery_b3c*****' where appid='encryption' and configkey='recoveryKeyId';

I've changed the following entry just to be sure (don't know if it's used by the new version) and deleted the remaining files_encryption entries.

mysql> update oc_appconfig set appid='encryption' where appid='files_encryption' and configkey='recoveryAdminEnabled';
mysql> delete from oc_appconfig where appid='files_encryption';

The migration script succeeded afterwards and all files (new and old) are accessible.

schiessle commented 9 years ago

@redhell do you have enabled server side encryption in your admin settings and loaded the "default encyption module" app? Just to double-check.

Can you also try to open the encrypted file with a normal text editor and post here what you see at the top between "HBEGIN" and "HEND"

schiessle commented 9 years ago

@twouters Thanks for your feedback and detailed explanation. You performed the right steps.

maste9 commented 9 years ago

I just received this failure message after deactivating trash and having the issue before:

[Doctrine\DBAL\Exception\UniqueConstraintViolationException]
An exception occurred while executing 'UPDATE oc_appconfig SET appid =
? WHERE appid = ?' with params ["encryption", "files_encryption"]:
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'encr
yption-publicShareKeyId' for key 'PRIMARY'

[Doctrine\DBAL\Driver\PDOException]
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'encr
yption-publicShareKeyId' for key 'PRIMARY'

[PDOException]
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'encr
yption-publicShareKeyId' for key 'PRIMARY'

encryption:migrate [user_id1] ... [user_idN]

orson74 commented 9 years ago

occ encryption:migrate

exit with error

[OC\ServerNotAvailableException] Lost connection to LDAP server.

if i try to re-run

[OCP\Files\LockNotAcquiredException] Could not obtain lock type 1 on "/var/www/owncloud/data/admin/files_encryption/admin.privateKey".

schiessle commented 9 years ago

@maste9 This is that issue: https://github.com/owncloud/core/issues/17443, fix is here: https://github.com/owncloud/core/pull/17473

redhell commented 9 years ago

Default Encryption is enabled: On a new file: HBEGIN:oc_encryption_module:OC_DEFAULT_MODULE:cipher:AES-256-CFB:HEND

The old files don't have that tag.

jtoloe commented 9 years ago

@schiesbn, @jknockaert OK, solved my issue. For some reason my users got new keys created (because migration failed and they logged in without the old keys being present?). After replacing the user privateKey and publicKey with copies from backup everything works (except that files created while the new keys were there are unreadable, a minor issue for me).

schiessle commented 9 years ago

@jtoloe thanks for the feedback, good to know that you could solve it. Probably you was in some intermediate state after the first migration try failed and the new encryption already started to work, create keys, etc

schiessle commented 9 years ago

@redhell That looks good. If no tag is available the encryption should fall back to the "default_encryption_module". At the moment I don't see why this shouldn't happen. I will have another look at the code and come back to you

jtoloe commented 9 years ago

@schiesbn it seems as if the publicKey is not copied while running the migration script after doing the manual database update. So the correct privateKey is present but without the publicKey. On login the privateKey gets overwritten with a new one (because the public key is missing?).

schiessle commented 9 years ago

fix for the trashbin issue is here: https://github.com/owncloud/core/pull/17474 combining this fix together with https://github.com/owncloud/core/pull/17473 (sql error)

If you are not that familiar with git, basically you need to replace two files:

/apps/encryption/lib/migration.php with https://github.com/owncloud/core/blob/enc_fix_migration/apps/encryption/lib/migration.php /apps/files_trashbin/lib/storage.php with https://github.com/owncloud/core/blob/fix_trashbin/apps/files_trashbin/lib/storage.php

double-check that you really downloaded the file content and not a html page or something similar.

phjr commented 9 years ago

I got those two files and ran occ encryption:migrate, I got several pages of html output and at the end "PHP Fatal error: Class 'OCA\Encryption\Migration' not found in /var/www/owncloud/apps/encryption/command/migratekeys.php on line 80 "