[Bug]: Nextcloud 29.0.3 upgrade failure "OC\ServerNotAvailableException: Could not decrypt key"

[thegreenbear]

⚠️ This issue respects the following points: ⚠️

• [X] This is a bug, not a question or a configuration/webserver/proxy issue.
• [X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• [X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• [X] I agree to follow Nextcloud's Code of Conduct.

Bug description

Upgrading from Nextcloud 29.0.2.2 to 29.0.3.4 fails with the following message:

OC\ServerNotAvailableException: Could not decrypt key

I am using the manual all-in-one deployment (it's been working great!) The containers tag used for version 29.0.2.2 was: 20240617_084300-latest The containers tag used for version 29.0.3.4 was: 20240701_074701-latest My set up is a bit special as I am using S3 as primary storage and encryption.

Note that I have 2 setups (one test, one prod) and both show the same behavior.

I have seen some issues on GitHub with the same error message but they seemed to apply to older versions of Nextcloud.

Steps to reproduce

1. Deploy nextcloud version 29.0.2 with the AIO manual method
2. Upgrade to 29.0.3
3. Bim

Note: my two setups were initially deployed with 28.x.x. I'm not sure if the earlier upgrade path is relevant or not. If it helps, I'm happy to test this specific point.

Expected behavior

Upgrade should work. Nextcloud should start.

Nextcloud Server version

29

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Updated from a MINOR version (ex. 28.0.1 to 28.0.2)

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

What user-backends are you using?

• [X] Default user-backend (database)
• [ ] LDAP/ Active Directory
• [ ] SSO - SAML
• [ ] Other

Configuration report

{
    "system": {
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "appsallowlist": false,
        "check_data_directory_permissions": false,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "***REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "***REMOVED SENSITIVE VALUE***",
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "region": "***REMOVED SENSITIVE VALUE***",
                "hostname": "***REMOVED SENSITIVE VALUE***",
                "port": "",
                "objectPrefix": "urn:oid:",
                "autocreate": true,
                "use_ssl": true,
                "use_path_style": false,
                "legacy_auth": false
            }
        },
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "29.0.2.2",
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "loglevel": "2",
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": "10485760",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "enabledPreviewProviders": {
            "1": "OC\\Preview\\Image",
            "2": "OC\\Preview\\MarkDown",
            "3": "OC\\Preview\\MP3",
            "4": "OC\\Preview\\TXT",
            "5": "OC\\Preview\\OpenDocument",
            "6": "OC\\Preview\\Movie",
            "7": "OC\\Preview\\Krita"
        },
        "enable_previews": true,
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": "30",
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "updatedirectory": "\/nc-updater",
        "maintenance_window_start": 100,
        "allow_local_remote_servers": true,
        "davstorage.request_timeout": 3600,
        "htaccess.RewriteBase": "\/",
        "dbpersistent": false,
        "auth.bruteforce.protection.enabled": true,
        "ratelimit.protection.enabled": true,
        "files_external_allow_create_new_local": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "default_phone_region": "CH",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "admin",
            "users"
        ],
        "twofactor_enforced_excluded_groups": []
    }
}

List of activated Apps

Enabled:
  - activity: 2.21.1
  - admin_audit: 1.19.0
  - calendar: 4.7.16
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - comments: 1.19.0
  - contacts: 6.0.0
  - contactsinteraction: 1.10.0
  - cookbook: 0.11.1
  - dashboard: 7.9.0
  - dav: 1.30.1
  - deck: 1.13.1
  - encryption: 2.17.0
  - external: 5.4.0
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - nextcloud_announcements: 1.18.0
  - notes: 4.10.1
  - notifications: 2.17.0
  - notify_push: 0.7.0
  - oauth2: 1.17.0
  - password_policy: 1.19.0
  - photos: 2.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - richdocuments: 8.4.5
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - support: 1.12.0
  - survey_client: 1.17.0
  - systemtags: 1.19.0
  - tasks: 0.16.1
  - text: 3.10.0
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0
Disabled:
  - bruteforcesettings: 2.9.0
  - files_external: 1.21.0
  - nextcloud-aio: 0.6.0
  - suspicious_login: 7.0.0
  - user_ldap: 1.20.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Upgrading nextcloud from 29.0.2.2 to 29.0.4.1...
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
System config value integrity.check.disabled deleted
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Setting log level to debug
Turned on maintenance mode
Updating database schema
Updated database
OC\ServerNotAvailableException: Could not decrypt key
Update failed
Maintenance mode is kept active
Resetting log level
Upgrade failed. Please restore from backup.
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Posting notifications to users that are admins...
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Posting 'Nextcloud update to 29.0.4.1 failed!' to: admin
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade

Also these seem relevant:

{
    "reqId": "HAem1UvoVjiIjlavfLlP",
    "level": 3,
    "time": "2024-08-31T20:25:37+00:00",
    "remoteAddr": "",
    "user": "--",
    "app": "objectstore",
    "method": "",
    "url": "--",
    "message": "Could not get object urn:oid:168 for file files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey",
    "userAgent": "--",
    "version": "29.0.2.2",
    "exception": {
        "Exception": "InvalidArgumentException",
        "Message": null,
        "Code": 0,
        "Trace": [
            {
                "file": "/var/www/html/3rdparty/guzzlehttp/psr7/src/MessageTrait.php",
                "line": 209,
                "function": "assertValue",
                "class": "GuzzleHttp\\Psr7\\Request",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "function": "GuzzleHttp\\Psr7\\{closure}",
                "class": "GuzzleHttp\\Psr7\\Request",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/guzzlehttp/psr7/src/MessageTrait.php",
                "line": 200,
                "function": "array_map",
                "args": [
                    [
                        "Closure"
                    ],
                    [
                        "*** sensitive parameters replaced ***"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/guzzlehttp/psr7/src/MessageTrait.php",
                "line": 174,
                "function": "trimAndValidateHeaderValues",
                "class": "GuzzleHttp\\Psr7\\Request",
                "type": "->",
                "args": [
                    [
                        "*** sensitive parameters replaced ***"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/guzzlehttp/psr7/src/MessageTrait.php",
                "line": 154,
                "function": "normalizeHeaderValue",
                "class": "GuzzleHttp\\Psr7\\Request",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/guzzlehttp/psr7/src/Request.php",
                "line": 49,
                "function": "setHeaders",
                "class": "GuzzleHttp\\Psr7\\Request",
                "type": "->",
                "args": [
                    [
                        "bytes=0-",
                        "AES256",
                        "",
                        "*** sensitive parameters replaced ***"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/Api/Serializer/RestSerializer.php",
                "line": 50,
                "function": "__construct",
                "class": "GuzzleHttp\\Psr7\\Request",
                "type": "->",
                "args": [
                    "GET",
                    [
                        "GuzzleHttp\\Psr7\\Uri"
                    ],
                    [
                        "bytes=0-",
                        "AES256",
                        "",
                        "*** sensitive parameters replaced ***"
                    ],
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/Middleware.php",
                "line": 96,
                "function": "__invoke",
                "class": "Aws\\Api\\Serializer\\RestSerializer",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/InputValidationMiddleware.php",
                "line": 73,
                "function": "Aws\\{closure}",
                "class": "Aws\\Middleware",
                "type": "::",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/Middleware.php",
                "line": 80,
                "function": "__invoke",
                "class": "Aws\\InputValidationMiddleware",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***",
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php",
                "line": 585,
                "function": "Aws\\{closure}",
                "class": "Aws\\Middleware",
                "type": "::",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php",
                "line": 608,
                "function": "Aws\\S3\\{closure}",
                "class": "Aws\\S3\\S3Client",
                "type": "::",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php",
                "line": 542,
                "function": "Aws\\S3\\{closure}",
                "class": "Aws\\S3\\S3Client",
                "type": "::",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php",
                "line": 561,
                "function": "Aws\\S3\\{closure}",
                "class": "Aws\\S3\\S3Client",
                "type": "::",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/Middleware.php",
                "line": 54,
                "function": "Aws\\S3\\{closure}",
                "class": "Aws\\S3\\S3Client",
                "type": "::",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/S3/SSECMiddleware.php",
                "line": 59,
                "function": "Aws\\{closure}",
                "class": "Aws\\Middleware",
                "type": "::",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/IdempotencyTokenMiddleware.php",
                "line": 77,
                "function": "__invoke",
                "class": "Aws\\S3\\SSECMiddleware",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***",
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "function": "__invoke",
                "class": "Aws\\IdempotencyTokenMiddleware",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/aws/aws-sdk-php/src/functions.php",
                "line": 363,
                "function": "call_user_func",
                "args": [
                    [
                        "Aws\\IdempotencyTokenMiddleware"
                    ],
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php",
                "line": 66,
                "function": "Aws\\serialize",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Files/Stream/SeekableHttpStream.php",
                "line": 93,
                "function": "OC\\Files\\ObjectStore\\{closure}",
                "class": "OC\\Files\\ObjectStore\\S3",
                "type": "->",
                "args": [
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Files/Stream/SeekableHttpStream.php",
                "line": 168,
                "function": "reconnect",
                "class": "OC\\Files\\Stream\\SeekableHttpStream",
                "type": "->",
                "args": [
                    0
                ]
            },
            {
                "function": "stream_open",
                "class": "OC\\Files\\Stream\\SeekableHttpStream",
                "type": "->",
                "args": [
                    "httpseek://",
                    "r",
                    [
                        [
                            "Closure"
                        ]
                    ],
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Files/Stream/SeekableHttpStream.php",
                "line": 67,
                "function": "fopen",
                "args": [
                    "httpseek://",
                    "r",
                    false,
                    null
                ]
            },
            {
                "file": "/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php",
                "line": 60,
                "function": "open",
                "class": "OC\\Files\\Stream\\SeekableHttpStream",
                "type": "::",
                "args": [
                    [
                        "Closure"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php",
                "line": 331,
                "function": "readObject",
                "class": "OC\\Files\\ObjectStore\\S3",
                "type": "->",
                "args": [
                    "urn:oid:168"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Files/Storage/Common.php",
                "line": 202,
                "function": "fopen",
                "class": "OC\\Files\\ObjectStore\\ObjectStoreStorage",
                "type": "->",
                "args": [
                    "files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey",
                    "r"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Files/View.php",
                "line": 1171,
                "function": "file_get_contents",
                "class": "OC\\Files\\Storage\\Common",
                "type": "->",
                "args": [
                    "files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Files/View.php",
                "line": 584,
                "function": "basicOperation",
                "class": "OC\\Files\\View",
                "type": "->",
                "args": [
                    "file_get_contents",
                    "/files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey",
                    [
                        "read"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/private/Encryption/Keys/Storage.php",
                "line": 272,
                "function": "file_get_contents",
                "class": "OC\\Files\\View",
                "type": "->",
                "args": [
                    "/files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Encryption/Keys/Storage.php",
                "line": 228,
                "function": "getKey",
                "class": "OC\\Encryption\\Keys\\Storage",
                "type": "->",
                "args": [
                    "/files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Encryption/Keys/Storage.php",
                "line": 120,
                "function": "getKeyWithUid",
                "class": "OC\\Encryption\\Keys\\Storage",
                "type": "->",
                "args": [
                    "/files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey",
                    "*** sensitive parameters replaced ***"
                ]
            },
            {
                "file": "/var/www/html/apps/encryption/lib/KeyManager.php",
                "line": 552,
                "function": "getSystemUserKey",
                "class": "OC\\Encryption\\Keys\\Storage",
                "type": "->",
                "args": [
                    "pubShare_1d023a1f.publicKey",
                    "OC_DEFAULT_MODULE"
                ]
            },
            {
                "file": "/var/www/html/apps/encryption/lib/KeyManager.php",
                "line": 96,
                "function": "getPublicShareKey",
                "class": "OCA\\Encryption\\KeyManager",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/apps/encryption/lib/Users/Setup.php",
                "line": 62,
                "function": "validateShareKey",
                "class": "OCA\\Encryption\\KeyManager",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/apps/encryption/lib/AppInfo/Application.php",
                "line": 56,
                "function": "setupSystem",
                "class": "OCA\\Encryption\\Users\\Setup",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/apps/encryption/appinfo/app.php",
                "line": 37,
                "function": "setUp",
                "class": "OCA\\Encryption\\AppInfo\\Application",
                "type": "->",
                "args": [
                    [
                        "OC\\Encryption\\Manager"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/private/App/AppManager.php",
                "line": 525,
                "args": [
                    "/var/www/html/apps/encryption/appinfo/app.php"
                ],
                "function": "require_once"
            },
            {
                "file": "/var/www/html/lib/private/App/AppManager.php",
                "line": 416,
                "function": "requireAppFile",
                "class": "OC\\App\\AppManager",
                "type": "::",
                "args": [
                    [
                        "OCA\\Encryption\\AppInfo\\Application"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/private/legacy/OC_App.php",
                "line": 139,
                "function": "loadApp",
                "class": "OC\\App\\AppManager",
                "type": "->",
                "args": [
                    "encryption"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Updater.php",
                "line": 362,
                "function": "loadApp",
                "class": "OC_App",
                "type": "::",
                "args": [
                    "encryption"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Updater.php",
                "line": 266,
                "function": "doAppUpgrade",
                "class": "OC\\Updater",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/lib/private/Updater.php",
                "line": 133,
                "function": "doUpgrade",
                "class": "OC\\Updater",
                "type": "->",
                "args": [
                    "29.0.3.4",
                    "29.0.2.2"
                ]
            },
            {
                "file": "/var/www/html/core/Command/Upgrade.php",
                "line": 216,
                "function": "upgrade",
                "class": "OC\\Updater",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/3rdparty/symfony/console/Command/Command.php",
                "line": 298,
                "function": "execute",
                "class": "OC\\Core\\Command\\Upgrade",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Console\\Input\\ArgvInput"
                    ],
                    [
                        "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/console/Application.php",
                "line": 1040,
                "function": "run",
                "class": "Symfony\\Component\\Console\\Command\\Command",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Console\\Input\\ArgvInput"
                    ],
                    [
                        "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/console/Application.php",
                "line": 301,
                "function": "doRunCommand",
                "class": "Symfony\\Component\\Console\\Application",
                "type": "->",
                "args": [
                    [
                        "OC\\Core\\Command\\Upgrade"
                    ],
                    [
                        "Symfony\\Component\\Console\\Input\\ArgvInput"
                    ],
                    [
                        "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/console/Application.php",
                "line": 171,
                "function": "doRun",
                "class": "Symfony\\Component\\Console\\Application",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Console\\Input\\ArgvInput"
                    ],
                    [
                        "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/private/Console/Application.php",
                "line": 213,
                "function": "run",
                "class": "Symfony\\Component\\Console\\Application",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Console\\Input\\ArgvInput"
                    ],
                    [
                        "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                    ]
                ]
            },
            {
                "file": "/var/www/html/console.php",
                "line": 102,
                "function": "run",
                "class": "OC\\Console\\Application",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/occ",
                "line": 11,
                "args": [
                    "/var/www/html/console.php"
                ],
                "function": "require_once"
            }
        ],
        "File": "/var/www/html/3rdparty/guzzlehttp/psr7/src/MessageTrait.php",
        "Line": 260,
        "message": "Could not get object urn:oid:168 for file files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey",
        "exception": {},
        "CustomMessage": "Could not get object urn:oid:168 for file files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey"
    }
}
{
    "reqId": "HAem1UvoVjiIjlavfLlP",
    "level": 3,
    "time": "2024-08-31T20:25:37+00:00",
    "remoteAddr": "",
    "user": "--",
    "app": "no app in context",
    "method": "",
    "url": "--",
    "message": "Could not decrypt key",
    "userAgent": "--",
    "version": "29.0.2.2",
    "exception": {
        "Exception": "OC\\ServerNotAvailableException",
        "Message": "Could not decrypt key",
        "Code": 0,
        "Trace": [
            {
                "file": "/var/www/html/lib/private/Encryption/Keys/Storage.php",
                "line": 228,
                "function": "getKey",
                "class": "OC\\Encryption\\Keys\\Storage",
                "type": "->",
                "args": [
                    "/files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Encryption/Keys/Storage.php",
                "line": 120,
                "function": "getKeyWithUid",
                "class": "OC\\Encryption\\Keys\\Storage",
                "type": "->",
                "args": [
                    "/files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey",
                    null
                ]
            },
            {
                "file": "/var/www/html/apps/encryption/lib/KeyManager.php",
                "line": 552,
                "function": "getSystemUserKey",
                "class": "OC\\Encryption\\Keys\\Storage",
                "type": "->",
                "args": [
                    "pubShare_1d023a1f.publicKey",
                    "OC_DEFAULT_MODULE"
                ]
            },
            {
                "file": "/var/www/html/apps/encryption/lib/KeyManager.php",
                "line": 96,
                "function": "getPublicShareKey",
                "class": "OCA\\Encryption\\KeyManager",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/apps/encryption/lib/Users/Setup.php",
                "line": 62,
                "function": "validateShareKey",
                "class": "OCA\\Encryption\\KeyManager",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/apps/encryption/lib/AppInfo/Application.php",
                "line": 56,
                "function": "setupSystem",
                "class": "OCA\\Encryption\\Users\\Setup",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/apps/encryption/appinfo/app.php",
                "line": 37,
                "function": "setUp",
                "class": "OCA\\Encryption\\AppInfo\\Application",
                "type": "->",
                "args": [
                    [
                        "OC\\Encryption\\Manager"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/private/App/AppManager.php",
                "line": 525,
                "args": [
                    "/var/www/html/apps/encryption/appinfo/app.php"
                ],
                "function": "require_once"
            },
            {
                "file": "/var/www/html/lib/private/App/AppManager.php",
                "line": 416,
                "function": "requireAppFile",
                "class": "OC\\App\\AppManager",
                "type": "::",
                "args": [
                    [
                        "OCA\\Encryption\\AppInfo\\Application"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/private/legacy/OC_App.php",
                "line": 139,
                "function": "loadApp",
                "class": "OC\\App\\AppManager",
                "type": "->",
                "args": [
                    "encryption"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Updater.php",
                "line": 362,
                "function": "loadApp",
                "class": "OC_App",
                "type": "::",
                "args": [
                    "encryption"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Updater.php",
                "line": 266,
                "function": "doAppUpgrade",
                "class": "OC\\Updater",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/lib/private/Updater.php",
                "line": 133,
                "function": "doUpgrade",
                "class": "OC\\Updater",
                "type": "->",
                "args": [
                    "29.0.3.4",
                    "29.0.2.2"
                ]
            },
            {
                "file": "/var/www/html/core/Command/Upgrade.php",
                "line": 216,
                "function": "upgrade",
                "class": "OC\\Updater",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/3rdparty/symfony/console/Command/Command.php",
                "line": 298,
                "function": "execute",
                "class": "OC\\Core\\Command\\Upgrade",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Console\\Input\\ArgvInput"
                    ],
                    [
                        "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/console/Application.php",
                "line": 1040,
                "function": "run",
                "class": "Symfony\\Component\\Console\\Command\\Command",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Console\\Input\\ArgvInput"
                    ],
                    [
                        "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/console/Application.php",
                "line": 301,
                "function": "doRunCommand",
                "class": "Symfony\\Component\\Console\\Application",
                "type": "->",
                "args": [
                    [
                        "OC\\Core\\Command\\Upgrade"
                    ],
                    [
                        "Symfony\\Component\\Console\\Input\\ArgvInput"
                    ],
                    [
                        "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/console/Application.php",
                "line": 171,
                "function": "doRun",
                "class": "Symfony\\Component\\Console\\Application",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Console\\Input\\ArgvInput"
                    ],
                    [
                        "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/private/Console/Application.php",
                "line": 213,
                "function": "run",
                "class": "Symfony\\Component\\Console\\Application",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Console\\Input\\ArgvInput"
                    ],
                    [
                        "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                    ]
                ]
            },
            {
                "file": "/var/www/html/console.php",
                "line": 102,
                "function": "run",
                "class": "OC\\Console\\Application",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/occ",
                "line": 11,
                "args": [
                    "/var/www/html/console.php"
                ],
                "function": "require_once"
            }
        ],
        "File": "/var/www/html/lib/private/Encryption/Keys/Storage.php",
        "Line": 285,
        "Previous": {
            "Exception": "Exception",
            "Message": "Authenticated ciphertext could not be decoded.",
            "Code": 0,
            "Trace": [
                {
                    "file": "/var/www/html/lib/private/Security/Crypto.php",
                    "line": 123,
                    "function": "decryptWithoutSecret",
                    "class": "OC\\Security\\Crypto",
                    "type": "->",
                    "args": [
                        "*** sensitive parameters replaced ***"
                    ]
                },
                {
                    "file": "/var/www/html/lib/private/Encryption/Keys/Storage.php",
                    "line": 283,
                    "function": "decrypt",
                    "class": "OC\\Security\\Crypto",
                    "type": "->",
                    "args": [
                        "*** sensitive parameters replaced ***"
                    ]
                },
                {
                    "file": "/var/www/html/lib/private/Encryption/Keys/Storage.php",
                    "line": 228,
                    "function": "getKey",
                    "class": "OC\\Encryption\\Keys\\Storage",
                    "type": "->",
                    "args": [
                        "/files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey"
                    ]
                },
                {
                    "file": "/var/www/html/lib/private/Encryption/Keys/Storage.php",
                    "line": 120,
                    "function": "getKeyWithUid",
                    "class": "OC\\Encryption\\Keys\\Storage",
                    "type": "->",
                    "args": [
                        "/files_encryption/OC_DEFAULT_MODULE/pubShare_1d023a1f.publicKey",
                        null
                    ]
                },
                {
                    "file": "/var/www/html/apps/encryption/lib/KeyManager.php",
                    "line": 552,
                    "function": "getSystemUserKey",
                    "class": "OC\\Encryption\\Keys\\Storage",
                    "type": "->",
                    "args": [
                        "pubShare_1d023a1f.publicKey",
                        "OC_DEFAULT_MODULE"
                    ]
                },
                {
                    "file": "/var/www/html/apps/encryption/lib/KeyManager.php",
                    "line": 96,
                    "function": "getPublicShareKey",
                    "class": "OCA\\Encryption\\KeyManager",
                    "type": "->",
                    "args": []
                },
                {
                    "file": "/var/www/html/apps/encryption/lib/Users/Setup.php",
                    "line": 62,
                    "function": "validateShareKey",
                    "class": "OCA\\Encryption\\KeyManager",
                    "type": "->",
                    "args": []
                },
                {
                    "file": "/var/www/html/apps/encryption/lib/AppInfo/Application.php",
                    "line": 56,
                    "function": "setupSystem",
                    "class": "OCA\\Encryption\\Users\\Setup",
                    "type": "->",
                    "args": []
                },
                {
                    "file": "/var/www/html/apps/encryption/appinfo/app.php",
                    "line": 37,
                    "function": "setUp",
                    "class": "OCA\\Encryption\\AppInfo\\Application",
                    "type": "->",
                    "args": [
                        [
                            "OC\\Encryption\\Manager"
                        ]
                    ]
                },
                {
                    "file": "/var/www/html/lib/private/App/AppManager.php",
                    "line": 525,
                    "args": [
                        "/var/www/html/apps/encryption/appinfo/app.php"
                    ],
                    "function": "require_once"
                },
                {
                    "file": "/var/www/html/lib/private/App/AppManager.php",
                    "line": 416,
                    "function": "requireAppFile",
                    "class": "OC\\App\\AppManager",
                    "type": "::",
                    "args": [
                        [
                            "OCA\\Encryption\\AppInfo\\Application"
                        ]
                    ]
                },
                {
                    "file": "/var/www/html/lib/private/legacy/OC_App.php",
                    "line": 139,
                    "function": "loadApp",
                    "class": "OC\\App\\AppManager",
                    "type": "->",
                    "args": [
                        "encryption"
                    ]
                },
                {
                    "file": "/var/www/html/lib/private/Updater.php",
                    "line": 362,
                    "function": "loadApp",
                    "class": "OC_App",
                    "type": "::",
                    "args": [
                        "encryption"
                    ]
                },
                {
                    "file": "/var/www/html/lib/private/Updater.php",
                    "line": 266,
                    "function": "doAppUpgrade",
                    "class": "OC\\Updater",
                    "type": "->",
                    "args": []
                },
                {
                    "file": "/var/www/html/lib/private/Updater.php",
                    "line": 133,
                    "function": "doUpgrade",
                    "class": "OC\\Updater",
                    "type": "->",
                    "args": [
                        "29.0.3.4",
                        "29.0.2.2"
                    ]
                },
                {
                    "file": "/var/www/html/core/Command/Upgrade.php",
                    "line": 216,
                    "function": "upgrade",
                    "class": "OC\\Updater",
                    "type": "->",
                    "args": []
                },
                {
                    "file": "/var/www/html/3rdparty/symfony/console/Command/Command.php",
                    "line": 298,
                    "function": "execute",
                    "class": "OC\\Core\\Command\\Upgrade",
                    "type": "->",
                    "args": [
                        [
                            "Symfony\\Component\\Console\\Input\\ArgvInput"
                        ],
                        [
                            "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                        ]
                    ]
                },
                {
                    "file": "/var/www/html/3rdparty/symfony/console/Application.php",
                    "line": 1040,
                    "function": "run",
                    "class": "Symfony\\Component\\Console\\Command\\Command",
                    "type": "->",
                    "args": [
                        [
                            "Symfony\\Component\\Console\\Input\\ArgvInput"
                        ],
                        [
                            "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                        ]
                    ]
                },
                {
                    "file": "/var/www/html/3rdparty/symfony/console/Application.php",
                    "line": 301,
                    "function": "doRunCommand",
                    "class": "Symfony\\Component\\Console\\Application",
                    "type": "->",
                    "args": [
                        [
                            "OC\\Core\\Command\\Upgrade"
                        ],
                        [
                            "Symfony\\Component\\Console\\Input\\ArgvInput"
                        ],
                        [
                            "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                        ]
                    ]
                },
                {
                    "file": "/var/www/html/3rdparty/symfony/console/Application.php",
                    "line": 171,
                    "function": "doRun",
                    "class": "Symfony\\Component\\Console\\Application",
                    "type": "->",
                    "args": [
                        [
                            "Symfony\\Component\\Console\\Input\\ArgvInput"
                        ],
                        [
                            "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                        ]
                    ]
                },
                {
                    "file": "/var/www/html/lib/private/Console/Application.php",
                    "line": 213,
                    "function": "run",
                    "class": "Symfony\\Component\\Console\\Application",
                    "type": "->",
                    "args": [
                        [
                            "Symfony\\Component\\Console\\Input\\ArgvInput"
                        ],
                        [
                            "Symfony\\Component\\Console\\Output\\ConsoleOutput"
                        ]
                    ]
                },
                {
                    "file": "/var/www/html/console.php",
                    "line": 102,
                    "function": "run",
                    "class": "OC\\Console\\Application",
                    "type": "->",
                    "args": []
                },
                {
                    "file": "/var/www/html/occ",
                    "line": 11,
                    "args": [
                        "/var/www/html/console.php"
                    ],
                    "function": "require_once"
                }
            ],
            "File": "/var/www/html/lib/private/Security/Crypto.php",
            "Line": 135
        },
        "message": "Could not decrypt key",
        "exception": {},
        "CustomMessage": "Could not decrypt key"
    }
}

Additional info

No response

[thegreenbear]

Interestingly, it seems Nextcloud searches for an object called urn:oid:168. But this object does not exists on the S3 storage. At least not after the rollback (as in, restore previous container volumes and revert to image containing 29.0.2).

[thegreenbear]

Hello there, Just to say I'm happy to help troubleshoot this issue if I can. I don't know much about NextCloud but I do have a dev env where I reproduced the issue. So I'm happy to test for whoever has ideas what to do :-D Cheers,