search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
27.39k stars 4.07k forks source link

[Bug]: S3 Primary Storage Hashing Error #39981

Closed thesethcarter closed 1 year ago

thesethcarter commented 1 year ago

⚠️ This issue respects the following points: ⚠️

Bug description

After upgrading to version 27 from 26 I have been unable to upload any files larger than 150mb with S3 primary storage. The following error is continually displayed in the logs:

[PHP] Error: Declaration of Aws\HashingStream::seek($offset, $whence = Aws\SEEK_SET) must be compatible with Psr\Http\Message\StreamInterface::seek(int $offset, int $whence = Psr\Http\Message\SEEK_SET): void at /var/www/vault/3rdparty/aws/aws-sdk-php/src/HashingStream.php#50

PUT /remote.php/dav/uploads/seth/4173833945/0000000000000005
from 47.195.31.55 by seth at 2023-08-20T18:19:52+00:00

Steps to reproduce

  1. Upload a file larger than 150mb using S3 as primary storage.

Expected behavior

Successful file upload.

Installation method

Community manual installation with Archive

Nextcloud Server version

27

Operating system

Ubuntu

PHP engine version

8.2

Web server

Nginx

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

Upgraded from a MAJOR version.

Are you using the Nextcloud Server Encryption module?

Encryption is disabled.

What user-backends are you using?

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "vault.carter.estate"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "27.1.0.1",
        "overwrite.cli.url": "https:\/\/vault.carter.estate",
        "htaccess.RewriteBase": "\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "3306",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "the-vault",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "s3.us-east-1.wasabisys.com",
                "port": 443,
                "use_ssl": true,
                "region": "us-east-1",
                "use_path_style": true
            }
        },
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "tls",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "maintenance_window_start": 1,
        "simpleSignUpLink.shown": false,
        "theme": "",
        "loglevel": 1,
        "versions_retention_obligation": "90, auto",
        "filelocking.enabled": true,
        "trashbin_retention_obligation": "180, 365",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "default_phone_region": "US",
        "updater.release.channel": "beta",
        "app_install_overwrite": [
            "gluusso",
            "previewgenerator",
            "ownbackup",
            "uploaddetails",
            "user_saml",
            "bruteforcesettings",
            "suspicious_login",
            "files_external_ipfs",
            "extract"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - activity: 2.19.0
  - admin_audit: 1.17.0
  - audioplayer: 3.4.0
  - bruteforcesettings: 2.7.0
  - camerarawpreviews: 0.8.2
  - cloud_federation_api: 1.10.0
  - contacts: 5.4.0-beta.2
  - contactsinteraction: 1.8.0
  - dav: 1.27.0
  - federatedfilesharing: 1.17.0
  - files: 1.22.0
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - groupfolders: 15.0.2
  - login_notes: 1.2.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - nextcloud_announcements: 1.16.0
  - notifications: 2.15.0
  - notify_push: 0.6.3
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - previewgenerator: 5.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - spreed: 17.1.0-rc.1
  - support: 1.10.0
  - survey_client: 1.15.0
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_email: 2.7.3
  - twofactor_nextcloud_notification: 3.7.0
  - twofactor_totp: 9.0.0
  - updatenotification: 1.17.0
  - user_saml: 5.2.2
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - comments: 1.17.0 (installed 1.9.0)
  - dashboard: 7.7.0 (installed 7.0.0)
  - encryption: 2.15.0
  - federation: 1.17.0 (installed 1.9.0)
  - files_external: 1.19.0
  - firstrunwizard: 2.16.0 (installed 2.8.0)
  - photos: 2.3.0 (installed 1.5.0)
  - text: 3.8.0 (installed 3.0.1)
  - user_ldap: 1.17.0 (installed 1.13.1)
  - user_status: 1.7.0 (installed 1.2.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

{"reqId":"WrJli0uApOr2oj7bDJIA","level":3,"time":"2023-08-20T16:10:59+00:00","remoteAddr":"47.195.31.55","user":"seth","app":"PHP","method":"PUT","url":"/remote.php/dav/uploads/seth/2234580768/0000000000000006","message":"Declaration of Aws\\HashingStream::seek($offset, $whence = Aws\\SEEK_SET) must be compatible with Psr\\Http\\Message\\StreamInterface::seek(int $offset, int $whence = Psr\\Http\\Message\\SEEK_SET): void at /var/www/vault/3rdparty/aws/aws-sdk-php/src/HashingStream.php#50","userAgent":"Mozilla/5.0 (Macintosh) mirall/3.9.2git (build 16965) (Nextcloud, osx-23.0.0 ClientArchitecture: arm64 OsArchitecture: arm64)","version":"27.1.0.1","data":{"app":"PHP"}}
{"reqId":"7AK6FTSaPYP1aDDH3NQl","level":3,"time":"2023-08-20T16:11:04+00:00","remoteAddr":"47.195.31.55","user":"seth","app":"PHP","method":"PUT","url":"/remote.php/dav/uploads/seth/2234580768/0000000000000006","message":"Declaration of Aws\\HashingStream::seek($offset, $whence = Aws\\SEEK_SET) must be compatible with Psr\\Http\\Message\\StreamInterface::seek(int $offset, int $whence = Psr\\Http\\Message\\SEEK_SET): void at /var/www/vault/3rdparty/aws/aws-sdk-php/src/HashingStream.php#50","userAgent":"Mozilla/5.0 (Macintosh) mirall/3.9.2git (build 16965) (Nextcloud, osx-23.0.0 ClientArchitecture: arm64 OsArchitecture: arm64)","version":"27.1.0.1","data":{"app":"PHP"}}
{"reqId":"GEJmbDPevEHQjugb1j7u","level":3,"time":"2023-08-20T16:11:10+00:00","remoteAddr":"47.195.31.55","user":"seth","app":"PHP","method":"PUT","url":"/remote.php/dav/uploads/seth/2234580768/0000000000000006","message":"Declaration of Aws\\HashingStream::seek($offset, $whence = Aws\\SEEK_SET) must be compatible with Psr\\Http\\Message\\StreamInterface::seek(int $offset, int $whence = Psr\\Http\\Message\\SEEK_SET): void at /var/www/vault/3rdparty/aws/aws-sdk-php/src/HashingStream.php#50","userAgent":"Mozilla/5.0 (Macintosh) mirall/3.9.2git (build 16965) (Nextcloud, osx-23.0.0 ClientArchitecture: arm64 OsArchitecture: arm64)","version":"27.1.0.1","data":{"app":"PHP"}}
{"reqId":"TcCJwSygEleBm23zqc65","level":3,"time":"2023-08-20T16:11:17+00:00","remoteAddr":"47.195.31.55","user":"seth","app":"PHP","method":"PUT","url":"/remote.php/dav/uploads/seth/2234580768/0000000000000006","message":"Declaration of Aws\\HashingStream::seek($offset, $whence = Aws\\SEEK_SET) must be compatible with Psr\\Http\\Message\\StreamInterface::seek(int $offset, int $whence = Psr\\Http\\Message\\SEEK_SET): void at /var/www/vault/3rdparty/aws/aws-sdk-php/src/HashingStream.php#50","userAgent":"Mozilla/5.0 (Macintosh) mirall/3.9.2git (build 16965) (Nextcloud, osx-23.0.0 ClientArchitecture: arm64 OsArchitecture: arm64)","version":"27.1.0.1","data":{"app":"PHP"}}

Additional info

No response

joshtrichards commented 1 year ago

Well, shoot. I think you've encountered this:

https://github.com/aws/aws-sdk-php/issues/2681

Oh, wait you're running the beta?

Oh, good. I think there will be time to fix this before formal release.

We need to bump aws-sdk-php to >= v3.269.1:

szaimen commented 1 year ago

Cc @come-nc @kesselb

thesethcarter commented 1 year ago

Initially encountered the issue in the Stable release channel. Attempted to correct it by moving to Beta, however; the error persists.

joshtrichards commented 1 year ago

That code has been the same across the most recent NC25/NC26/NC27.

Your report lead me to believe this worked for you in NC26. Is that correct?

If so, which version NC26 of specifically worked previously?

thesethcarter commented 1 year ago

As far as I can remember everything worked without issue when running 26.0.5.

joshtrichards commented 1 year ago

Can you show me the line declaring the seek() function from your /var/www/vault/3rdparty/psr/http-message/src/StreamInterface.php?

It'll either be:

public function seek($offset, $whence = SEEK_SET);

or

public function seek(int $offset, int $whence = SEEK_SET): void;

Your error suggests is the latter, but that's not what is shown in our code base:

thesethcarter commented 1 year ago

It is currently:

public function seek($offset, $whence = SEEK_SET);

joshtrichards commented 1 year ago

It's the camerarawpreviews app:

https://github.com/ariselseng/camerarawpreviews/issues/97

thesethcarter commented 1 year ago

FIXED! As soon as I disabled it everything started uploading without an issue. Thank you!

joshtrichards commented 1 year ago

Awesome!

I think due to #37891 we should probably still bump up the aws-sdk-php version since I think it opened the door to psr/http-message being bumped to 2.0:

https://github.com/nextcloud/server/blob/0bb4577237e0c5f65d747a2b0ad68ac249b67907/build/integration/composer.lock#L553

Which the existing version of aws-sdk-php we use won't work with.

But composer dependencies are not my specialty so I'm simply noting this for the future for someone else to find. :)

ariselseng commented 1 year ago

If anybody has any tips on how I can avoid this problem in my app camerarawpreviews, it would be much appreciated. Is there a best practice here about using composer in nexcloud apps which I am not following or something? For instance, is there a way in my composer autoloader to namespace everything so that there is no global redeclaration of classes?

kesselb commented 1 year ago

Hi @ariselseng :wave:

We recently added an article about the dependency hell to our docs: https://docs.nextcloud.com/server/latest/developer_manual/app_development/dependency_management.html

intervention/image pulls in guzzlehttp/psr7 as dependency, but install a version that is incompatible with the copy shipped by Nextcloud: https://github.com/Intervention/image/blob/54934ae8ea3661fd189437df90fb09ec3b679c74/composer.json#L17

If you require psr/http-message:^1.1 in your composer.json it should install the right version. Even better would to drop intervention/image dependency.

We should update the aws-sdk in Nextcloud. But we usually don't back port dependency updates, and thus a newer version will not be shipped before Nextcloud 28.

ariselseng commented 1 year ago

Hi @ariselseng :wave:

We recently added an article about the dependency hell to our docs: https://docs.nextcloud.com/server/latest/developer_manual/app_development/dependency_management.html

intervention/image pulls in guzzlehttp/psr7 as dependency, but install a version that is incompatible with the copy shipped by Nextcloud: https://github.com/Intervention/image/blob/54934ae8ea3661fd189437df90fb09ec3b679c74/composer.json#L17

If you require psr/http-message:^1.1 in your composer.json it should install the right version. Even better would to drop intervention/image dependency.

We should update the aws-sdk in Nextcloud. But we usually don't back port dependency updates, and thus a newer version will not be shipped before Nextcloud 28.

Thank you for your insight. I will try to get rid of my dependencies. I wonder if it is possible to somehow patch the namespaces in the local vendor folder with a prefix or something, so that er can avoid all this together. Or tie the local composer with the core's composer so that the local composer is aware of where to find core's dependencies.

joshtrichards commented 1 year ago

@ariselseng I'm afraid your last comment is going to get lost since this issue has been closed out. Would you mind opening a dedicated issue with your comment? It's a fair question (and one that I don't feel qualified to address, but I'll do my best to direct to other qualified project participants). Feel free to ref this issue as the latest field example of this situation arising in the new issue you create.