[Bug]: Can't copy files from a folder to another when using S3 object storage as a Primary Storage

arturondc37 commented 2 years ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] Nextcloud Server is running on 64bit capable CPU, PHP and OS.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

After installation I uploaded a file and I can't copy it to another folder. But I can do the following operations without errors:

Move Files
Delete Files
Rename Files
Upload Files I'm using the S3 object storage provided by linode. I have the correct values of key and secret and the correct permision in the object storage.

Steps to reproduce

Take an official image of nextcloud docker
Configure a S3 object storage as a Primary Storage
Upload a file
Try to copy it to another folder

Expected behavior

Being able to copy files from one folder to another without problems.

Installation method

Official Docker image

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.0

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "hiperdata",
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "region": "us-east-1",
                "hostname": "hiperdata.us-east-1.linodeobjects.com",
                "port": "443",
                "objectPrefix": "urn:oid:",
                "autocreate": false,
                "use_ssl": true,
                "use_path_style": true,
                "legacy_auth": false
            }
        },
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "workspaces.hiperlogistica.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "24.0.3.2",
        "overwrite.cli.url": "http:\/\/workspaces.hiperlogistica.com",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true
    }
}

List of activated Apps

Enabled:
  - accessibility: 1.10.0
  - activity: 2.16.0
  - bruteforcesettings: 2.4.0
  - circles: 24.0.0
  - cloud_federation_api: 1.7.0
  - contactsinteraction: 1.5.0
  - dav: 1.22.0
  - federatedfilesharing: 1.14.0
  - federation: 1.14.0
  - files: 1.19.0
  - files_pdfviewer: 2.5.0
  - files_rightclick: 1.3.0
  - files_sharing: 1.16.2
  - files_trashbin: 1.14.0
  - files_versions: 1.17.0
  - files_videoplayer: 1.13.0
  - logreader: 2.9.0
  - lookup_server_connector: 1.12.0
  - nextcloud_announcements: 1.13.0
  - notifications: 2.12.0
  - oauth2: 1.12.0
  - photos: 1.6.0
  - privacy: 1.8.0
  - provisioning_api: 1.14.0
  - serverinfo: 1.14.0
  - settings: 1.6.0
  - sharebymail: 1.14.0
  - systemtags: 1.14.0
  - text: 3.5.1
  - theming: 1.15.0
  - twofactor_backupcodes: 1.13.0
  - updatenotification: 1.14.0
  - user_status: 1.4.0
  - viewer: 1.8.0
  - workflowengine: 2.6.0
Disabled:
  - admin_audit
  - comments: 1.14.0
  - dashboard: 7.4.0
  - encryption
  - files_external
  - firstrunwizard: 2.13.0
  - password_policy: 1.14.0
  - recommendations: 1.3.0
  - support: 1.7.0
  - survey_client: 1.12.0
  - user_ldap
  - weather_status: 1.4.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

[webdav] Fatal: Aws\S3\Exception\S3Exception: Error executing "CopyObject" on "https://hiperdata.us-east-1.linodeobjects.com/hiperdata/urn%3Aoid%3A376"; AWS HTTP error: Client error: `PUT https://hiperdata.us-east-1.linodeobjects.com/hiperdata/urn%3Aoid%3A376` resulted in a `404 Not Found` response:
<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>hiperdata</BucketName><RequestId>tx000000 (truncated...)
 NoSuchKey (client):  - <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>hiperdata</BucketName><RequestId>tx00000000000001aea0f28-0062f308fe-224d08cc-default</RequestId><HostId>224d08cc-default-default</HostId></Error> at <<closure>>

 0. /var/www/html/3rdparty/aws/aws-sdk-php/src/WrappedHttpHandler.php line 97
    Aws\WrappedHttpHandler->parseError("*** sensitive parameters replaced ***", GuzzleHttp\Psr7\Request {}, Aws\Command {}, [])
 1. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 204
    Aws\WrappedHttpHandler->Aws\{closure}("*** sensitive parameters replaced ***")
 2. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 169
    GuzzleHttp\Promise\Promise::callHandler(2, "*** sensitive parameters replaced ***", null)
 3. /var/www/html/3rdparty/guzzlehttp/promises/src/RejectedPromise.php line 42
    GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
 4. /var/www/html/3rdparty/guzzlehttp/promises/src/TaskQueue.php line 48
    GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
 5. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 158
    GuzzleHttp\Promise\TaskQueue->run()
 6. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 183
    GuzzleHttp\Handler\CurlMultiHandler->tick()
 7. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 248
    GuzzleHttp\Handler\CurlMultiHandler->execute("*** sensitive parameters replaced ***")
 8. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 224
    GuzzleHttp\Promise\Promise->invokeWaitFn()
 9. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
10. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
11. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 269
    GuzzleHttp\Promise\Promise->waitIfPending()
12. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 226
    GuzzleHttp\Promise\Promise->invokeWaitList()
13. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 62
    GuzzleHttp\Promise\Promise->waitIfPending()
14. /var/www/html/3rdparty/guzzlehttp/promises/src/Coroutine.php line 67
    GuzzleHttp\Promise\Promise->wait()
15. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 248
    GuzzleHttp\Promise\Coroutine->GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
16. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 224
    GuzzleHttp\Promise\Promise->invokeWaitFn()
17. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 62
    GuzzleHttp\Promise\Promise->waitIfPending()
18. /var/www/html/3rdparty/guzzlehttp/promises/src/Coroutine.php line 103
    GuzzleHttp\Promise\Promise->wait("*** sensitive parameters replaced ***")
19. /var/www/html/3rdparty/aws/aws-sdk-php/src/S3/S3ClientTrait.php line 63
    GuzzleHttp\Promise\Coroutine->wait()
20. /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php line 176
    Aws\S3\S3Client->copy("hiperdata", "urn:oid:364", "hiperdata", "urn:oid:376")
21. /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php line 607
    OC\Files\ObjectStore\S3->copyObject("urn:oid:364", "urn:oid:376")
22. /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php line 589
    OC\Files\ObjectStore\ObjectStoreStorage->copyFile(OC\Files\Cache\CacheEntry {}, "files/photo_2022-08-08_10-45-52.jpg")
23. /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php line 571
    OC\Files\ObjectStore\ObjectStoreStorage->copyInner(OC\Files\Cache\CacheEntry {}, "files/photo_2022-08-08_10-45-52.jpg")
24. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 290
    OC\Files\ObjectStore\ObjectStoreStorage->copy("files/Documents ... g", "files/photo_2022-08-08_10-45-52.jpg")
25. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 290
    OC\Files\Storage\Wrapper\Wrapper->copy("files/Documents ... g", "files/photo_2022-08-08_10-45-52.jpg")
26. /var/www/html/lib/private/Files/Storage/Wrapper/Availability.php line 304
    OC\Files\Storage\Wrapper\Wrapper->copy("files/Documents ... g", "files/photo_2022-08-08_10-45-52.jpg")
27. /var/www/html/lib/private/Files/View.php line 940
    OC\Files\Storage\Wrapper\Availability->copy("files/Documents ... g", "files/photo_2022-08-08_10-45-52.jpg")
28. /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php line 486
    OC\Files\View->copy("/Documents/phot ... g", "/photo_2022-08-08_10-45-52.jpg")
29. /var/www/html/3rdparty/sabre/dav/lib/DAV/Tree.php line 132
    OCA\DAV\Connector\Sabre\Directory->copyInto("photo_2022-08-08_10-45-52.jpg", "/Documents/phot ... g", OCA\DAV\Connector\Sabre\File {})
30. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 655
    Sabre\DAV\Tree->copy("files/admin/Doc ... g", "files/admin/pho ... g")
31. /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
    Sabre\DAV\CorePlugin->httpCopy(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
32. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 472
    Sabre\DAV\Server->emit("method:COPY", [Sabre\HTTP\Requ ... }])
33. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 253
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
34. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 321
    Sabre\DAV\Server->start()
35. /var/www/html/apps/dav/lib/Server.php line 352
    Sabre\DAV\Server->exec()
36. /var/www/html/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
37. /var/www/html/remote.php line 166
    require_once("/var/www/html/a ... p")

COPY /remote.php/dav/files/admin/Documents/photo_2022-08-08_10-45-52.jpg
from 143.198.96.24 by admin at 2022-08-10T01:25:18+00:00

Additional info

No response

szaimen commented 1 year ago

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

arturodc37 commented 1 year ago

@szaimen I upgraded my nextcloud installation to 25.0.3 and the issue persist. Also as a side effect document versioning is not working, in fact, from the previous version it didn't work but I thought the problem was something with the installation but in this version it doesn't work either. In block storage everything works perfectly. Maybe is a Linode issue with Object Storage Backend. I'm going to try using an Amazon object storage.

arturodc37 commented 1 year ago

@szaimen Problem Solved, It seems the problem is related to the Object storage URL in Linode S3. When a bucket is created in Linode the bucket address is, for example: mybucket.region.linodeobjects.com But the above causes, in the case of Nextcloud, that it creates a folder inside the bucket with the name of the bucket. To avoid this problem, region.linodeobjects.com should be used as the URL because the bucket name is specified in a separate variable. Finally the correct syntax would be like this: Example for environment in docker: -OBJECTSTORE_S3_HOST=us-southeast-1.linodeobjects.com -OBJECTSTORE_S3_BUCKET=mybucket -OBJECTSTORE_S3_KEY=mykey -OBJECTSTORE_S3_SECRET=mysecret -OBJECTSTORE_S3_PORT=443 -OBJECTSTORE_S3_SSL=true -OBJECTSTORE_S3_REGION=us-southeast-1 -OBJECTSTORE_S3_USEPATH_STYLE=true

But hey, the above apparently is not a problem of the Linode object storage but of the way in which Nextcloud manages those configurations.

So this issue can be marked as solved

szaimen commented 1 year ago

So is this a docker variable problem?

arturodc37 commented 1 year ago

@szaimen No, it's not a docker variable problem. It is the way in which each provider presents its url to access the bucket. In the specific case of Linode, when you create a bucket, the URL represents the direct access to the content inside the bucket and not the root directory. Making an analogy is as if I created a folder in /home/ called example, the path to the folder would be /home/example but the path I really need is /home/ because in that root directory I will be able to see the example folder. So it's not a problem, it's just knowing these details because each provider has its way of managing its object storage.

szaimen commented 1 year ago

All right, so I think this should potentially be documented here? https://docs.nextcloud.com Moving to the correct repo then.

joshtrichards commented 1 year ago

Your issue was two-fold:

you used the direct bucket URL instead of the provider's S3 endpoint hostname
you had use_path_style enabled when it wasn't necessary

The docs technically covered this before, but were a bit confusing. That's been remedied some recently, but I can see already see there's room for further improvement. Namely to make sure it is clear that the hostname (aka: OBJECTSTORE_S3_HOST for Docker) is meant to be the S3 endpoint, not the direct bucket URL.

https://www.linode.com/docs/products/storage/object-storage/guides/urls/#cluster-url-s3-endpoint

As for use_path_style (aka: OBJECTSTORE_S3_USEPATH_STYLE):

You may need to use use_path_style if your non-Amazon S3 store does not support requests like https://bucket.hostname.domain/. Setting use_path_style to true configures the S3 client to make requests like https://hostname.domain/bucket instead.

Often both styles will work with some providers. The main issue is that the hostname must only ever be the S3 endpoint.

Documenting all of this clearly is also challenging because a few other configuration combinations will technically sometimes work, but they won't get the results you might expect or may be fragile (i.e. break in the future).

It gets confusing very quickly!

I tried to narrow down the choices recently in the latest docs to the two reliable combinations that always (to my knowledge) work and always place your data where you're expecting (regardless of provider/object store platform)

https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html#simple-storage-service-s3

From the looks of it, Linode would work either as:

hostname: us-southeast-1.linodeobjects.com
bucket: mybucket
use_path_style: true

With the above resulting in your data being stored in https://us-southeast-1.linodeobjects.com/mybucket/

or:

hostname: us-southeast-1.linodeobjects.com
bucket: mybucket
use_path_style: false

With the above resulting in your data being stored in https://mybucket.us-southeast-1.linodeobjects.com/

I'll see about:

Adding language for "S3 endpoint" to the explanation text for the hostname parameter in the NC Admin Manual. And possibly even language to make it clear we don't want the direct bucket URL, if one is provided.
The community Docker README language could probably use an update as well, but it also at least links to these Admin Manual further on down in the S3 variables section.

nextcloud / documentation