[Bug]: Files larger ~4GB fail when using Backblze B2 as Object Store on Nextcloud Hub II (23.0.6)

Worble commented 2 years ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

Uploading a file larger than 4GB when using Backblaze B2 as object store on Nextcloud Hub II (23.0.6) causes the error Sabre\DAV\Exception: Stream from assembly node shorter than expected, got 28747089 bytes, expected 96889854.

Full exception:

    Sabre\DAV\Exception: Stream from assembly node shorter than expected, got 28747089 bytes, expected 96889854

    /var/www/html/apps/dav/lib/Connector/Sabre/File.php - line 316:

    OCA\DAV\Connector\Sabre\File->convertToSabreException(Exception {})

    /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php - line 155:

    OCA\DAV\Connector\Sabre\File->put(null)

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php - line 307:

    OCA\DAV\Connector\Sabre\Directory->createFile("File nam ... p", null)

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php - line 133:

    Sabre\DAV\Tree->copyNode(OCA\DAV\Upload\FutureFile {}, OCA\DAV\Conn ... {}, "File nam ... p")

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php - line 163:

    Sabre\DAV\Tree->copy("uploads/wor ... e", "files/worbl ... p")

    /var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php - line 94:

    Sabre\DAV\Tree->move("uploads/wor ... e", "files/worbl ... p")

    /var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php - line 76:

    OCA\DAV\Upload\ChunkingPlugin->performMove("uploads/wor ... e", "files/worbl ... p")

    /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php - line 89:

    OCA\DAV\Upload\ChunkingPlugin->beforeMove("uploads/wor ... e", "files/worbl ... p")

    /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 603:

    Sabre\DAV\Server->emit("beforeMove", [ "uploads/w ... "])

    /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php - line 89:

    Sabre\DAV\CorePlugin->httpMove(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php - line 472:

    Sabre\DAV\Server->emit("method:MOVE", [ Sabre\HTTP ... }])

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php - line 253:

    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php - line 321:

    Sabre\DAV\Server->start()

    /var/www/html/apps/dav/lib/Server.php - line 339:

    Sabre\DAV\Server->exec()

    /var/www/html/apps/dav/appinfo/v2/remote.php - line 35:

    OCA\DAV\Server->exec()

    /var/www/html/remote.php - line 166:

    require_once("/var/www/ht ... p")

Caused by Exception: Stream from assembly node shorter than expected, got 28747089 bytes, expected 96889854

    <<closure>>

    OCA\DAV\Upload\AssemblyStream->stream_read(8192)

    /var/www/html/3rdparty/icewind/streams/src/Wrapper.php - line 55:

    fread(null, 8192)

    /var/www/html/3rdparty/icewind/streams/src/CallbackWrapper.php - line 96:

    Icewind\Streams\Wrapper->stream_read(8192)

    <<closure>>

    Icewind\Streams\CallbackWrapper->stream_read(8192)

    /var/www/html/3rdparty/icewind/streams/src/Wrapper.php - line 55:

    fread(null, 8192)

    /var/www/html/3rdparty/icewind/streams/src/CountWrapper.php - line 87:

    Icewind\Streams\Wrapper->stream_read(8192)

    <<closure>>

    Icewind\Streams\CountWrapper->stream_read(8192)

    /var/www/html/3rdparty/guzzlehttp/psr7/src/Stream.php - line 228:

    fread(null, 8192)

    /var/www/html/3rdparty/guzzlehttp/psr7/src/AppendStream.php - line 205:

    GuzzleHttp\Psr7\Stream->read(8192)

    /var/www/html/3rdparty/guzzlehttp/psr7/src/LimitStream.php - line 152:

    GuzzleHttp\Psr7\AppendStream->read(8192)

    /var/www/html/3rdparty/aws/aws-sdk-php/src/HashingStream.php - line 38:

    GuzzleHttp\Psr7\LimitStream->read(8192)

    /var/www/html/3rdparty/guzzlehttp/psr7/src/Utils.php - line 53:

    Aws\HashingStream->read(8192)

    /var/www/html/3rdparty/aws/aws-sdk-php/src/S3/MultipartUploader.php - line 117:

    GuzzleHttp\Psr7\Utils::copyToStream(Aws\HashingS ... }}, GuzzleHttp\Psr7\Stream {})

    /var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploader.php - line 54:

    Aws\S3\MultipartUploader->createPart(false, 6)

    /var/www/html/3rdparty/aws/aws-sdk-php/src/CommandPool.php - line 55:

    Aws\Multipart\AbstractUploader->getUploadCommands(Closure {})

    <<closure>>

    Aws\CommandPool->Aws\{closure}("*** sensiti ... *")

    /var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php - line 212:

    Generator->next()

    /var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php - line 238:

    GuzzleHttp\Promise\EachPromise->advanceIterator()

    /var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php - line 183:

    GuzzleHttp\Promise\EachPromise->step(4)

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 204:

    GuzzleHttp\Promise\EachPromise->GuzzleHttp\Promise\{closure}("*** sensiti ... *")

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 153:

    GuzzleHttp\Promise\Promise::callHandler(1, "*** sensiti ... *", null)

    /var/www/html/3rdparty/guzzlehttp/promises/src/TaskQueue.php - line 48:

    GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}("*** sensiti ... *")

    /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php - line 158:

    GuzzleHttp\Promise\TaskQueue->run()

    /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php - line 183:

    GuzzleHttp\Handler\CurlMultiHandler->tick()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 248:

    GuzzleHttp\Handler\CurlMultiHandler->execute("*** sensiti ... *")

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 224:

    GuzzleHttp\Promise\Promise->invokeWaitFn()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 269:

    GuzzleHttp\Promise\Promise->waitIfPending()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 226:

    GuzzleHttp\Promise\Promise->invokeWaitList()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 269:

    GuzzleHttp\Promise\Promise->waitIfPending()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 226:

    GuzzleHttp\Promise\Promise->invokeWaitList()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 62:

    GuzzleHttp\Promise\Promise->waitIfPending()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Coroutine.php - line 67:

    GuzzleHttp\Promise\Promise->wait()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 248:

    GuzzleHttp\Promise\Coroutine->GuzzleHttp\Promise\{closure}("*** sensiti ... *")

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 224:

    GuzzleHttp\Promise\Promise->invokeWaitFn()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 269:

    GuzzleHttp\Promise\Promise->waitIfPending()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 226:

    GuzzleHttp\Promise\Promise->invokeWaitList()

    /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php - line 62:

    GuzzleHttp\Promise\Promise->waitIfPending()

    /var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php - line 83:

    GuzzleHttp\Promise\Promise->wait()

    /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php - line 121:

    Aws\Multipart\AbstractUploadManager->upload()

    /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php - line 154:

    OC\Files\ObjectStore\S3->writeMultiPart("urn:oid:97178", GuzzleHttp\Psr7\AppendStream {}, "application/octet-stream")

    /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php - line 491:

    OC\Files\ObjectStore\S3->writeObject("urn:oid:97178", null, "application/octet-stream")

    /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php - line 647:

    OC\Files\ObjectStore\ObjectStoreStorage->writeStream("files/Backu ... p", null, null)

    /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php - line 647:

    OC\Files\Storage\Wrapper\Wrapper->writeStream("files/Backu ... p", null, null)

    /var/www/html/apps/dav/lib/Connector/Sabre/File.php - line 245:

    OC\Files\Storage\Wrapper\Wrapper->writeStream("files/Backu ... p", null)

    /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php - line 155:

    OCA\DAV\Connector\Sabre\File->put(null)

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php - line 307:

    OCA\DAV\Connector\Sabre\Directory->createFile("File nam ... p", null)

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php - line 133:

    Sabre\DAV\Tree->copyNode(OCA\DAV\Upload\FutureFile {}, OCA\DAV\Conn ... {}, "File nam ... p")

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php - line 163:

    Sabre\DAV\Tree->copy("uploads/wor ... e", "files/worbl ... p")

    /var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php - line 94:

    Sabre\DAV\Tree->move("uploads/wor ... e", "files/worbl ... p")

    /var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php - line 76:

    OCA\DAV\Upload\ChunkingPlugin->performMove("uploads/wor ... e", "files/worbl ... p")

    /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php - line 89:

    OCA\DAV\Upload\ChunkingPlugin->beforeMove("uploads/wor ... e", "files/worbl ... p")

    /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 603:

    Sabre\DAV\Server->emit("beforeMove", [ "uploads/w ... "])

    /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php - line 89:

    Sabre\DAV\CorePlugin->httpMove(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php - line 472:

    Sabre\DAV\Server->emit("method:MOVE", [ Sabre\HTTP ... }])

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php - line 253:

    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})

    /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php - line 321:

    Sabre\DAV\Server->start()

    /var/www/html/apps/dav/lib/Server.php - line 339:

    Sabre\DAV\Server->exec()

    /var/www/html/apps/dav/appinfo/v2/remote.php - line 35:

    OCA\DAV\Server->exec()

    /var/www/html/remote.php - line 166:

    require_once("/var/www/ht ... p")

Steps to reproduce

Add a file larger than 4gb
Wait for it to upload
Observe error after it has been fully uploaded

Expected behavior

File to be uploaded successfully.

Installation method

Other

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "overwritehost": "nextcloud.worble.xyz",
        "overwriteprotocol": "https",
        "overwrite.cli.url": "https:\/\/nextcloud.worble.xyz\/",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "nextcloud-df4xPkCikEP9pK",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "s3.eu-central-003.backblazeb2.com",
                "port": 443,
                "use_ssl": true,
                "use_path_style": false
            }
        },
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cryptic-pig.pikapod.net"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "23.0.6.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_smtpauth": 1,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "ssl",
        "loglevel": 2,
        "maintenance": false,
        "filelocking.enabled": false,
        "filesystem_check_changes": 0
    }
}

List of activated Apps

Enabled:
  - accessibility: 1.9.0
  - activity: 2.15.0
  - breezedark: 23.2.2
  - bruteforcesettings: 2.4.0
  - calendar: 3.4.0
  - circles: 23.1.2
  - cloud_federation_api: 1.6.0
  - comments: 1.13.0
  - contacts: 4.1.1
  - contactsinteraction: 1.4.0
  - dashboard: 7.3.0
  - dav: 1.21.0
  - deck: 1.6.3
  - federatedfilesharing: 1.13.0
  - federation: 1.13.0
  - files: 1.18.0
  - files_pdfviewer: 2.4.0
  - files_rightclick: 1.2.0
  - files_sharing: 1.15.0
  - files_trashbin: 1.13.0
  - files_versions: 1.16.0
  - files_videoplayer: 1.12.0
  - firstrunwizard: 2.12.0
  - logreader: 2.8.0
  - lookup_server_connector: 1.11.0
  - mail: 1.13.5
  - maps: 0.1.10
  - music: 1.5.2
  - news: 18.1.0
  - nextcloud_announcements: 1.12.0
  - notes: 4.3.1
  - notifications: 2.11.1
  - oauth2: 1.11.0
  - occweb: 0.1.0
  - password_policy: 1.13.0
  - photos: 1.5.0
  - privacy: 1.7.0
  - provisioning_api: 1.13.0
  - recommendations: 1.2.0
  - richdocuments: 5.0.5
  - richdocumentscode: 21.11.402
  - serverinfo: 1.13.0
  - settings: 1.5.0
  - sharebymail: 1.13.0
  - spreed: 13.0.6
  - support: 1.6.0
  - survey_client: 1.11.0
  - systemtags: 1.13.0
  - text: 3.4.1
  - theming: 1.14.0
  - twofactor_backupcodes: 1.12.0
  - twofactor_totp: 6.4.0
  - twofactor_webauthn: 0.3.1
  - updatenotification: 1.13.0
  - user_status: 1.3.1
  - viewer: 1.7.0
  - weather_status: 1.3.0
  - workflowengine: 2.5.0
Disabled:
  - admin_audit
  - encryption
  - files_external
  - user_ldap

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

These logs contain a lot of sensitive information in regards to folder/file names, can these be stripped somehow?

Additional info

No response

Worble commented 2 years ago

Is there anything else I can provide or do to help with this?

FALErwi commented 2 years ago

I'm facing same issue with s3 object store. There are another issues when using Object storage as primary. 1- Can't upload files with large size. 2- Can't remove the files from the object store even if the file doesn't exist in the nextcloud instance. 3- Slow performance.

szaimen commented 1 year ago

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

Worble commented 1 year ago

Hi, I can confirm this is still happening on 25.0.3. Configuration, installed apps, etc, should all still be the same as above.

Most recent error log:

[webdav] Error: Sabre\DAV\Exception: Unable to read from stream at <<closure>>

 0. /var/www/html/apps/dav/lib/Connector/Sabre/File.php line 318
    OCA\DAV\Connector\Sabre\File->convertToSabreException(RuntimeException {})
 1. /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php line 151
    OCA\DAV\Connector\Sabre\File->put(null)
 2. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 307
    OCA\DAV\Connector\Sabre\Directory->createFile("[filename] ... v", null)
 3. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 133
    Sabre\DAV\Tree->copyNode(OCA\DAV\Upload\FutureFile {}, OCA\DAV\Connector\Sabre\Directory {}, "[filename] ... v")
 4. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 163
    Sabre\DAV\Tree->copy("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
 5. /var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php line 94
    Sabre\DAV\Tree->move("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
 6. /var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php line 76
    OCA\DAV\Upload\ChunkingPlugin->performMove("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
 7. /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
    OCA\DAV\Upload\ChunkingPlugin->beforeMove("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
 8. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 603
    Sabre\DAV\Server->emit("beforeMove", ["uploads/[user] ... "])
 9. /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
    Sabre\DAV\CorePlugin->httpMove(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
10. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 472
    Sabre\DAV\Server->emit("method:MOVE", [Sabre\HTTP\Requ ... }])
11. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 253
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
12. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 321
    Sabre\DAV\Server->start()
13. /var/www/html/apps/dav/lib/Server.php line 360
    Sabre\DAV\Server->exec()
14. /var/www/html/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
15. /var/www/html/remote.php line 171
    require_once("/var/www/html/a ... p")

MOVE /remote.php/dav/uploads/[user]/2402750380/.file
from [ip address] by [user] at 2023-01-23T20:25:38+00:00

[no app in context] Error: RuntimeException: Unable to read from stream at <<closure>>

 0. /var/www/html/3rdparty/guzzlehttp/psr7/src/AppendStream.php line 203
    GuzzleHttp\Psr7\Stream->read(8192)
 1. /var/www/html/3rdparty/guzzlehttp/psr7/src/LimitStream.php line 152
    GuzzleHttp\Psr7\AppendStream->read(8192)
 2. /var/www/html/3rdparty/aws/aws-sdk-php/src/HashingStream.php line 38
    GuzzleHttp\Psr7\LimitStream->read(8192)
 3. /var/www/html/3rdparty/guzzlehttp/psr7/src/Utils.php line 53
    Aws\HashingStream->read(8192)
 4. /var/www/html/3rdparty/aws/aws-sdk-php/src/S3/MultipartUploader.php line 117
    GuzzleHttp\Psr7\Utils::copyToStream(Aws\HashingStrea ... }}, GuzzleHttp\Psr7\Stream {})
 5. /var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploader.php line 54
    Aws\S3\MultipartUploader->createPart(false, 4)
 6. /var/www/html/3rdparty/aws/aws-sdk-php/src/CommandPool.php line 56
    Aws\Multipart\AbstractUploader->getUploadCommands(Closure {})
 7. <<closure>>
    Aws\CommandPool->Aws\{closure}("*** sensitive parameters replaced ***")
 8. /var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php line 205
    Generator->next()
 9. /var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php line 149
    GuzzleHttp\Promise\EachPromise->advanceIterator()
10. /var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php line 82
    GuzzleHttp\Promise\EachPromise->refillPending()
11. /var/www/html/3rdparty/aws/aws-sdk-php/src/CommandPool.php line 80
    GuzzleHttp\Promise\EachPromise->promise()
12. /var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php line 132
    Aws\CommandPool->promise()
13. <<closure>>
    Aws\Multipart\AbstractUploadManager->Aws\Multipart\{closure}("*** sensitive parameters replaced ***")
14. /var/www/html/3rdparty/guzzlehttp/promises/src/Coroutine.php line 140
    Generator->send(Aws\Result {})
15. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 204
    GuzzleHttp\Promise\Coroutine->_handleSuccess(Aws\Result {})
16. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 153
    GuzzleHttp\Promise\Promise::callHandler(1, Aws\Result {}, null)
17. /var/www/html/3rdparty/guzzlehttp/promises/src/TaskQueue.php line 48
    GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
18. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 159
    GuzzleHttp\Promise\TaskQueue->run()
19. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 184
    GuzzleHttp\Handler\CurlMultiHandler->tick()
20. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 248
    GuzzleHttp\Handler\CurlMultiHandler->execute("*** sensitive parameters replaced ***")
21. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 224
    GuzzleHttp\Promise\Promise->invokeWaitFn()
22. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
23. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
24. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
25. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
26. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 62
    GuzzleHttp\Promise\Promise->waitIfPending()
27. /var/www/html/3rdparty/guzzlehttp/promises/src/Coroutine.php line 67
    GuzzleHttp\Promise\Promise->wait()
28. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 248
    GuzzleHttp\Promise\Coroutine->GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
29. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 224
    GuzzleHttp\Promise\Promise->invokeWaitFn()
30. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
31. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
32. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 62
    GuzzleHttp\Promise\Promise->waitIfPending()
33. /var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php line 83
    GuzzleHttp\Promise\Promise->wait()
34. /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php line 129
    Aws\Multipart\AbstractUploadManager->upload()
35. /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php line 162
    OC\Files\ObjectStore\S3->writeMultiPart("urn:oid:302458", GuzzleHttp\Psr7\AppendStream {}, "video/x-matroska")
36. /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php line 491
    OC\Files\ObjectStore\S3->writeObject("urn:oid:302458", null, "video/x-matroska")
37. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 647
    OC\Files\ObjectStore\ObjectStoreStorage->writeStream("files/[folder]/ ... v", null, null)
38. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 647
    OC\Files\Storage\Wrapper\Wrapper->writeStream("files/[folder]/ ... v", null, null)
39. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 647
    OC\Files\Storage\Wrapper\Wrapper->writeStream("files/[folder]/ ... v", null, null)
40. /var/www/html/apps/dav/lib/Connector/Sabre/File.php line 248
    OC\Files\Storage\Wrapper\Wrapper->writeStream("files/[folder]/ ... v", null)
41. /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php line 151
    OCA\DAV\Connector\Sabre\File->put(null)
42. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 307
    OCA\DAV\Connector\Sabre\Directory->createFile("[filename] ... v", null)
43. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 133
    Sabre\DAV\Tree->copyNode(OCA\DAV\Upload\FutureFile {}, OCA\DAV\Connector\Sabre\Directory {}, "[filename] ... v")
44. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 163
    Sabre\DAV\Tree->copy("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
45. /var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php line 94
    Sabre\DAV\Tree->move("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
46. /var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php line 76
    OCA\DAV\Upload\ChunkingPlugin->performMove("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
47. /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
    OCA\DAV\Upload\ChunkingPlugin->beforeMove("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
48. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 603
    Sabre\DAV\Server->emit("beforeMove", ["uploads/[user] ... "])
49. /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
    Sabre\DAV\CorePlugin->httpMove(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
50. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 472
    Sabre\DAV\Server->emit("method:MOVE", [Sabre\HTTP\Requ ... }])
51. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 253
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
52. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 321
    Sabre\DAV\Server->start()
53. /var/www/html/apps/dav/lib/Server.php line 360
    Sabre\DAV\Server->exec()
54. /var/www/html/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
55. /var/www/html/remote.php line 171
    require_once("/var/www/html/a ... p")

MOVE /remote.php/dav/uploads/[user]/2402750380/.file
from [ip address] by [user] at 2023-01-23T20:25:38+00:00

[objectstore] Error: RuntimeException: Unable to read from stream at <<closure>>

 0. /var/www/html/3rdparty/guzzlehttp/psr7/src/AppendStream.php line 203
    GuzzleHttp\Psr7\Stream->read(8192)
 1. /var/www/html/3rdparty/guzzlehttp/psr7/src/LimitStream.php line 152
    GuzzleHttp\Psr7\AppendStream->read(8192)
 2. /var/www/html/3rdparty/aws/aws-sdk-php/src/HashingStream.php line 38
    GuzzleHttp\Psr7\LimitStream->read(8192)
 3. /var/www/html/3rdparty/guzzlehttp/psr7/src/Utils.php line 53
    Aws\HashingStream->read(8192)
 4. /var/www/html/3rdparty/aws/aws-sdk-php/src/S3/MultipartUploader.php line 117
    GuzzleHttp\Psr7\Utils::copyToStream(Aws\HashingStrea ... }}, GuzzleHttp\Psr7\Stream {})
 5. /var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploader.php line 54
    Aws\S3\MultipartUploader->createPart(false, 4)
 6. /var/www/html/3rdparty/aws/aws-sdk-php/src/CommandPool.php line 56
    Aws\Multipart\AbstractUploader->getUploadCommands(Closure {})
 7. <<closure>>
    Aws\CommandPool->Aws\{closure}("*** sensitive parameters replaced ***")
 8. /var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php line 205
    Generator->next()
 9. /var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php line 149
    GuzzleHttp\Promise\EachPromise->advanceIterator()
10. /var/www/html/3rdparty/guzzlehttp/promises/src/EachPromise.php line 82
    GuzzleHttp\Promise\EachPromise->refillPending()
11. /var/www/html/3rdparty/aws/aws-sdk-php/src/CommandPool.php line 80
    GuzzleHttp\Promise\EachPromise->promise()
12. /var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php line 132
    Aws\CommandPool->promise()
13. <<closure>>
    Aws\Multipart\AbstractUploadManager->Aws\Multipart\{closure}("*** sensitive parameters replaced ***")
14. /var/www/html/3rdparty/guzzlehttp/promises/src/Coroutine.php line 140
    Generator->send(Aws\Result {})
15. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 204
    GuzzleHttp\Promise\Coroutine->_handleSuccess(Aws\Result {})
16. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 153
    GuzzleHttp\Promise\Promise::callHandler(1, Aws\Result {}, null)
17. /var/www/html/3rdparty/guzzlehttp/promises/src/TaskQueue.php line 48
    GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
18. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 159
    GuzzleHttp\Promise\TaskQueue->run()
19. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 184
    GuzzleHttp\Handler\CurlMultiHandler->tick()
20. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 248
    GuzzleHttp\Handler\CurlMultiHandler->execute("*** sensitive parameters replaced ***")
21. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 224
    GuzzleHttp\Promise\Promise->invokeWaitFn()
22. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
23. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
24. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
25. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
26. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 62
    GuzzleHttp\Promise\Promise->waitIfPending()
27. /var/www/html/3rdparty/guzzlehttp/promises/src/Coroutine.php line 67
    GuzzleHttp\Promise\Promise->wait()
28. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 248
    GuzzleHttp\Promise\Coroutine->GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
29. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 224
    GuzzleHttp\Promise\Promise->invokeWaitFn()
30. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
31. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
32. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 62
    GuzzleHttp\Promise\Promise->waitIfPending()
33. /var/www/html/3rdparty/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php line 83
    GuzzleHttp\Promise\Promise->wait()
34. /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php line 129
    Aws\Multipart\AbstractUploadManager->upload()
35. /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php line 162
    OC\Files\ObjectStore\S3->writeMultiPart("urn:oid:302458", GuzzleHttp\Psr7\AppendStream {}, "video/x-matroska")
36. /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php line 491
    OC\Files\ObjectStore\S3->writeObject("urn:oid:302458", null, "video/x-matroska")
37. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 647
    OC\Files\ObjectStore\ObjectStoreStorage->writeStream("files/[folder]/ ... v", null, null)
38. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 647
    OC\Files\Storage\Wrapper\Wrapper->writeStream("files/[folder]/ ... v", null, null)
39. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 647
    OC\Files\Storage\Wrapper\Wrapper->writeStream("files/[folder]/ ... v", null, null)
40. /var/www/html/apps/dav/lib/Connector/Sabre/File.php line 248
    OC\Files\Storage\Wrapper\Wrapper->writeStream("files/[folder]/ ... v", null)
41. /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php line 151
    OCA\DAV\Connector\Sabre\File->put(null)
42. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 307
    OCA\DAV\Connector\Sabre\Directory->createFile("[filename] ... v", null)
43. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 133
    Sabre\DAV\Tree->copyNode(OCA\DAV\Upload\FutureFile {}, OCA\DAV\Connector\Sabre\Directory {}, "[filename] ... v")
44. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 163
    Sabre\DAV\Tree->copy("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
45. /var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php line 94
    Sabre\DAV\Tree->move("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
46. /var/www/html/apps/dav/lib/Upload/ChunkingPlugin.php line 76
    OCA\DAV\Upload\ChunkingPlugin->performMove("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
47. /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
    OCA\DAV\Upload\ChunkingPlugin->beforeMove("uploads/[user]/2402750380/.file", "files/[user]/Vi ... v")
48. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 603
    Sabre\DAV\Server->emit("beforeMove", ["uploads/[user] ... "])
49. /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
    Sabre\DAV\CorePlugin->httpMove(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
50. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 472
    Sabre\DAV\Server->emit("method:MOVE", [Sabre\HTTP\Requ ... }])
51. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 253
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
52. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 321
    Sabre\DAV\Server->start()
53. /var/www/html/apps/dav/lib/Server.php line 360
    Sabre\DAV\Server->exec()
54. /var/www/html/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
55. /var/www/html/remote.php line 171
    require_once("/var/www/html/a ... p")

MOVE /remote.php/dav/uploads/[user]/2402750380/.file
from [ip address] by [user] at 2023-01-23T20:25:38+00:00

joshtrichards commented 1 year ago

@Worble Can you provide the raw log entry for one of these failed transfers? The formatted ones don't include the previous Throwable. Thanks! (P.S. I haven't been able to replicate this using B2/S3 as an External Storage, but I haven't tried it as Primary yet so take that with a grain of salt).

@FALErwi You should open separate issues for your other two experiences. And, if you think your first one is related to this one, include your matching exception. If it doesn't match, open up a dedicated issue for it too!

nextcloud-command commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

joshtrichards commented 10 months ago

@Worble This Issue auto-closed yesterday awaiting raw log info, but I re-opened it to remind myself to test more fully. I did so this morning and I cannot reproduce this behavior (in current NC27.1). I'm testing against a B2 bucket configured as Primary Storage with >4GB files. I was never able to reproduce this previously either.

I suspect this is a config issue if you're still encountering it so I'm going to re-close this for now. Feel free to follow-up with the raw logs from the upload period if you're still experiencing this and I'll re-look.

nextcloud / server