nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.89k stars 4.01k forks source link

[Bug]: Unable to rename files residing in external storage (S3) #41804

Open datenangebot opened 10 months ago

datenangebot commented 10 months ago

⚠️ This issue respects the following points: ⚠️

Bug description

After upgrading to 27.1.3, we discovered that it is not possible anymore to rename files that reside in external S3 storage.

Exception from the server:
{"Exception":"Aws\\Exception\\MultipartUploadException","Message":"An exception occurred while uploading parts to a multipart upload. The following parts had errors:\n- Part 1: Error executing \"UploadPartCopy\" on \"https://???/test2.odt?partNumber=1&uploadId=2~Ifrvg95tAnlyFxa75hfrysf0stGe9Wb\"; AWS HTTP error: Client error: `PUT https://???/test2.odt?partNumber=1&uploadId=2~Ifrvg95tAnlyFxa75hfrysf0stGe9Wb` resulted in a `404 Not Found` response:\n<?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>NoSuchKey</Code><BucketName>???</BucketName><Reque (truncated...)\n NoSuchKey (client):  - <?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>NoSuchKey</Code><BucketName>???</BucketName><RequestId>tx00000e4cce185faa47f8f-006565dcde-f59e029-default</RequestId><HostId>???</HostId></Error>\n",

Steps to reproduce

Install NC27.1.3 with S3 storage.

Expected behavior

Can rename

Installation method

None

Nextcloud Server version

27

Operating system

None

PHP engine version

None

Web server

None

Database engine version

None

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

Configuration report

No response

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

Seems to be related to: https://github.com/nextcloud/server/pull/40577 If we revert this patch, it works fine.

mickenordin commented 10 months ago

I went through and disabled all apps one by one to confirm that it was not related to guzzle being vendored twice like I have seen for some other folks. And reverting the patch indeed fixes the issue. In our case Ceph is used to provide S3 storage.

This is the exact patch I used to revert the multi part upload on 27.1.4.2 if someone is interested

diff --git a/apps/files_external/lib/Lib/Storage/AmazonS3.php b/apps/files_external/lib/Lib/Storage/AmazonS3.php
index 8e2a6b8ea12..5619b7be8dd 100644
--- a/apps/files_external/lib/Lib/Storage/AmazonS3.php
+++ b/apps/files_external/lib/Lib/Storage/AmazonS3.php
@@ -582,7 +582,10 @@ class AmazonS3 extends \OC\Files\Storage\Common {

        if ($isFile === true || $this->is_file($source)) {
            try {
-               $this->copyObject($source, $target, [
+               $this->getConnection()->copyObject([
+                   'Bucket' => $this->bucket,
+                   'Key' => $this->cleanKey($target),
+                   'CopySource' => S3Client::encodeKey($this->bucket . '/' . $source),
                    'StorageClass' => $this->storageClass,
                ]);
                $this->testTimeout();
diff --git a/lib/private/Files/ObjectStore/S3ObjectTrait.php b/lib/private/Files/ObjectStore/S3ObjectTrait.php
index 217e1a1a2ff..7328287acee 100644
--- a/lib/private/Files/ObjectStore/S3ObjectTrait.php
+++ b/lib/private/Files/ObjectStore/S3ObjectTrait.php
@@ -189,23 +189,9 @@ trait S3ObjectTrait {
    public function objectExists($urn) {
        return $this->getConnection()->doesObjectExist($this->bucket, $urn, $this->getSSECParameters());
    }
-
-   public function copyObject($from, $to, array $options = []) {
-       $sourceMetadata = $this->getConnection()->headObject([
-           'Bucket' => $this->getBucket(),
-           'Key' => $from,
-       ] + $this->getSSECParameters());
-
-       $copy = new MultipartCopy($this->getConnection(), [
-           "source_bucket" => $this->getBucket(),
-           "source_key" => $from
-       ], array_merge([
-           "bucket" => $this->getBucket(),
-           "key" => $to,
-           "acl" => "private",
-           "params" => $this->getSSECParameters() + $this->getSSECParameters(true),
-           "source_metadata" => $sourceMetadata
-       ], $options));
-       $copy->copy();
+   public function copyObject($from, $to) {
+       $this->getConnection()->copy($this->getBucket(), $from, $this->getBucket(), $to, 'private', [
+           'params' => $this->getSSECParameters() + $this->getSSECParameters(true)
+       ]);
    }
 }