[Bug]: S3 Primary Storage broken after PHP upgrade to 8.3

lewisgardner commented 2 days ago

⚠️ This issue respects the following points: ⚠️

[x] This is a bug, not a question or a configuration/webserver/proxy issue.
[x] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[x] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[x] I agree to follow Nextcloud's Code of Conduct.

Bug description

After upgrading Ubuntu from 22.04 to 24.04 (which also upgrades PHP from 8.1 to 8.3) the AWS S3 ObjectStore is unable to connect to S3 compatible backend (DigitalOcean Spaces). The S3 ObjectStore is configured as the primary storage. The server worked perfectly with this setup before the OS (and PHP) upgrade, no other changes were performed.

I have downloaded latest AWS PHP SDK into test folder and performed connection tests from same Apache/PHP server and connection works correctly.

Steps to reproduce

1 .Run a nextcloud server with primary storage attached to S3 compatible backend

Upgrade server to PHP 8.3
Server can no longer access primary storage backend

Expected behavior

The nextcloud server should continue to be able to access the backend.

Nextcloud Server version

30

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.3

Web server

Apache (supported)

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

[x] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "default_phone_region": "AU",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "30.0.1.2",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_smtpmode": "smtp",
        "theme": "",
        "maintenance": false,
        "forcessl": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "loglevel": 3,
        "trashbin_retention_obligation": "auto",
        "updatechecker": false,
        "updater.release.channel": "stable",
        "mail_smtpauthtype": "PLAIN",
        "mail_smtpsecure": "ssl",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/www.myserver.com\/nextcloud",
        "htaccess.RewriteBase": "\/nextcloud",
        "mail_sendmailmode": "smtp",
        "mysql.utf8mb4": true,
        "maintenance_window_start": 15,
        "objectstore": {
            "class": "OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "mynextcloud",
                "autocreate": false,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "sgp1.digitaloceanspaces.com",
                "port": 443,
                "use_ssl": true,
                "region": "**region**",
                "use_path_style": false
            }
        }
    }
}

List of activated Apps

Enabled:
  - activity: 3.0.0
  - app_api: 4.0.0
  - bookmarks: 14.2.6
  - circles: 30.0.0-dev
  - cloud_federation_api: 1.13.0
  - comments: 1.20.1
  - contactsinteraction: 1.11.0
  - dashboard: 7.10.0
  - dav: 1.31.1
  - federatedfilesharing: 1.20.0
  - federation: 1.20.0
  - files: 2.2.0
  - files_downloadlimit: 3.0.0
  - files_pdfviewer: 3.0.0
  - files_reminders: 1.3.0
  - files_sharing: 1.22.0
  - files_trashbin: 1.20.1
  - files_versions: 1.23.0
  - firstrunwizard: 3.0.0
  - logreader: 3.0.0
  - lookup_server_connector: 1.18.0
  - nextcloud_announcements: 2.0.0
  - notifications: 3.0.0
  - oauth2: 1.18.1
  - password_policy: 2.0.0
  - photos: 3.0.2
  - privacy: 2.0.0
  - provisioning_api: 1.20.0
  - recommendations: 3.0.0
  - related_resources: 1.5.0
  - serverinfo: 2.0.0
  - settings: 1.13.0
  - sharebymail: 1.20.0
  - support: 2.0.0
  - survey_client: 2.0.0
  - systemtags: 1.20.0
  - text: 4.1.0
  - theming: 2.5.0
  - twofactor_backupcodes: 1.19.0
  - updatenotification: 1.20.0
  - user_status: 1.10.0
  - viewer: 3.0.0
  - weather_status: 1.10.0
  - webhook_listeners: 1.1.0-dev
  - workflowengine: 2.12.0
Disabled:
  - admin_audit: 1.20.0
  - bruteforcesettings: 3.0.0 (installed 2.4.0)
  - encryption: 2.18.0
  - files_external: 1.22.0
  - files_rightclick: 0.15.1 (installed 1.6.0)
  - suspicious_login: 8.0.0
  - twofactor_nextcloud_notification: 4.0.0
  - twofactor_totp: 12.0.0-dev
  - user_ldap: 1.21.0

Nextcloud Signing status

No login possible

Nextcloud Logs

{"reqId":"mL6wzUXm30UbRX9KHn8b","level":3,"time":"2024-10-19T06:40:03+00:00","remoteAddr":"","user":"--","app":"core","method":"","url":"--","message":"Error while running background job OCA\\Theming\\Jobs\\RestoreBackgroundImageColor (id: 5815, arguments: {\"stage\":\"prepare\"})","userAgent":"--","version":"30.0.1.2","exception":{"Exception":"Aws\\Exception\\UnresolvedEndpointException","Message":"Invalid region: region was not a valid DNS name.","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/EndpointV2/Rule/TreeRule.php","line":42,"function":"evaluate","class":"Aws\\EndpointV2\\Rule\\ErrorRule","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/EndpointV2/Rule/TreeRule.php","line":42,"function":"evaluate","class":"Aws\\EndpointV2\\Rule\\TreeRule","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/EndpointV2/Rule/TreeRule.php","line":42,"function":"evaluate","class":"Aws\\EndpointV2\\Rule\\TreeRule","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/EndpointV2/Rule/TreeRule.php","line":42,"function":"evaluate","class":"Aws\\EndpointV2\\Rule\\TreeRule","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/EndpointV2/Ruleset/Ruleset.php","line":68,"function":"evaluate","class":"Aws\\EndpointV2\\Rule\\TreeRule","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/EndpointV2/EndpointProviderV2.php","line":52,"function":"evaluate","class":"Aws\\EndpointV2\\Ruleset\\Ruleset","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/EndpointV2/EndpointV2Middleware.php","line":92,"function":"resolveEndpoint","class":"Aws\\EndpointV2\\EndpointProviderV2","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/Auth/AuthSelectionMiddleware.php","line":97,"function":"__invoke","class":"Aws\\EndpointV2\\EndpointV2Middleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/InputValidationMiddleware.php","line":73,"function":"__invoke","class":"Aws\\Auth\\AuthSelectionMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/Middleware.php","line":90,"function":"__invoke","class":"Aws\\InputValidationMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php","line":627,"function":"Aws\\{closure}","class":"Aws\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php","line":650,"function":"Aws\\S3\\{closure}","class":"Aws\\S3\\S3Client","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php","line":584,"function":"Aws\\S3\\{closure}","class":"Aws\\S3\\S3Client","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php","line":603,"function":"Aws\\S3\\{closure}","class":"Aws\\S3\\S3Client","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/Middleware.php","line":58,"function":"Aws\\S3\\{closure}","class":"Aws\\S3\\S3Client","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/SSECMiddleware.php","line":59,"function":"Aws\\{closure}","class":"Aws\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/IdempotencyTokenMiddleware.php","line":77,"function":"__invoke","class":"Aws\\S3\\SSECMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php","line":64,"function":"__invoke","class":"Aws\\IdempotencyTokenMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php","line":58,"function":"executeAsync","class":"Aws\\AwsClient","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/S3ClientTrait.php","line":363,"function":"execute","class":"Aws\\AwsClient","type":"->"},{"file":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/S3ClientTrait.php","line":272,"function":"checkExistenceWithCommand","class":"Aws\\S3\\S3Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/ObjectStore/S3ConnectionTrait.php","line":131,"function":"doesBucketExist","class":"Aws\\S3\\S3Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/ObjectStore/S3ObjectTrait.php","line":90,"function":"getConnection","class":"OC\\Files\\ObjectStore\\S3","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/ObjectStore/S3ObjectTrait.php","line":152,"function":"writeSingle","class":"OC\\Files\\ObjectStore\\S3","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php","line":521,"function":"writeObject","class":"OC\\Files\\ObjectStore\\S3","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php","line":468,"function":"writeStream","class":"OC\\Files\\ObjectStore\\ObjectStoreStorage","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":1134,"function":"file_put_contents","class":"OC\\Files\\ObjectStore\\ObjectStoreStorage","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":644,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Node/Folder.php","line":167,"function":"file_put_contents","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/SimpleFS/NewSimpleFile.php","line":103,"function":"newFile","class":"OC\\Files\\Node\\Folder","type":"->"},{"file":"/var/www/nextcloud/apps/theming/lib/Jobs/RestoreBackgroundImageColor.php","line":172,"function":"putContent","class":"OC\\Files\\SimpleFS\\NewSimpleFile","type":"->"},{"file":"/var/www/nextcloud/apps/theming/lib/Jobs/RestoreBackgroundImageColor.php","line":80,"function":"storeUserIdsToProcess","class":"OCA\\Theming\\Jobs\\RestoreBackgroundImageColor","type":"->"},{"file":"/var/www/nextcloud/apps/theming/lib/Jobs/RestoreBackgroundImageColor.php","line":50,"function":"runPreparation","class":"OCA\\Theming\\Jobs\\RestoreBackgroundImageColor","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/Job.php","line":61,"function":"run","class":"OCA\\Theming\\Jobs\\RestoreBackgroundImageColor","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php","line":43,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php","line":29,"function":"start","class":"OCP\\BackgroundJob\\QueuedJob","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":162,"function":"execute","class":"OCP\\BackgroundJob\\QueuedJob","type":"->"}],"File":"/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src/EndpointV2/Rule/ErrorRule.php","Line":41,"message":"Error while running background job OCA\\Theming\\Jobs\\RestoreBackgroundImageColor (id: 5815, arguments: {\"stage\":\"prepare\"})","exception":{},"CustomMessage":"Error while running background job OCA\\Theming\\Jobs\\RestoreBackgroundImageColor (id: 5815, arguments: {\"stage\":\"prepare\"})"}

Additional info

No response

SystemKeeper commented 2 days ago

Invalid region: region was not a valid DNS name.

Sounds like something you wanna check.

joshtrichards commented 2 days ago

"region": "**region**",

Is this what your region is actually set to?

Since you're not using AWS, you should be able to leave region entirely blank in your case since we already handle the scenario mentioned in DO's docs. Otherwise this must be set to - as strange as it sounds but like DO's own docs say - a valid AWS region. But like I said, blank is easier/best since we already handle this.

But this isn't new behavior.

lewisgardner commented 2 days ago

Thank you for the responses. I removed the 'region' parameter entirely and it works now. Please note that this is changed behaviour: The region parameter has been set to '**region**' ever since I created the server as the DigitalOcean documentation states that the region is ignored when using their S3 compatible storage. The only change I did was upgrade server from Ubuntu 22.04 to 24.04 and only after that I had to remove the 'region' parameter.

solracsf commented 1 day ago

Then I would considered tjis a bug fixed, not a bug introduced, becasue it should never work as is.

joshtrichards commented 1 day ago

Please note that this is changed behaviour: The region parameter has been set to 'region' ever since I created the server as the DigitalOcean documentation states that the region is ignored when using their S3 compatible storage

It's a little confusing, but the DO docs state it must be set to a valid AWS region. They're own example says us-east-1:

"Due to an AWS-specific behavior in all versions of the SDK except Python 3, to successfully create a new bucket, you must specify an AWS region, such as us-east-1, in your configuration. This is because, when creating a bucket, the SDK sends an entirely different payload if a custom region is specified, which results in an error."

We haven't changed our handling of the region parameter in many years. I'm not sure how your config ever worked.

Perhaps an earlier version of the AWS SDK ignored your value of **region** silently (since it's also happens to contain characters that aren't valid within DNS entries or something). That wouldn't have been reasonable behavior either by the SDK, so still a bug fixed. :) Also just a wild guess. I did a quick look and I don't see any obviously related changes in the SDK.

What version of Nextcloud did you upgrade from? (I'm presuming you upgraded Nextcloud too recently since 30.0.1 just came out).

lewisgardner commented 1 day ago

According to DO docs, the region only has to be valid AWS region when creating a bucket (which was not required in this case as I had created the bucket manually beforehand).

I was running version NC 29 when I upgraded the server from Ubuntu 22.04 to 24.04, after which the S3 connectivity was broken. I then updated to NC 30.0 (using update/updater.phar which still worked fine) and then 30.0.1 a few days later when it was released in case that would fix the issue. My best guess is that parameter handling has somehow changed between PHP 8.1 and 8.3 (they did introduce sensitive parameters in 8.2) but I'm just happy it all works now.

nextcloud / server