[Bug]: Syncing to S3 Storage (e.g. minio server) syncs folders twice into a subtree

Section1977 commented 10 months ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

look below

Steps to reproduce

1.Minio S3 Storage provide a bucket

Syncing to bucket with NExtcloud 27 syncs folders twice into a subtree
means /example plus content of folder syncs to /example plus content of folder AND /example/example plus content of folder
you see the bug only on the S3 storage side, not within Nextcloud !
pydio client e.g. does not make this bug in S3 structure, so it is a bug of nextcloud

Expected behavior

syncs only once with mirroring exactly the folder structure

Installation method

Community Docker image

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

None

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

No response

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

No response

joshtrichards commented 10 months ago

What do you mean by "syncing"? Please describe how, specifically, you have integrated S3 into your Nextcloud environment.

E.g.

As External Storage: https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage/amazons3.html
As Primary Storage: https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html

It would also be helpful if you fill out the entire issue template as requested.

Section1977 commented 10 months ago

hi, minio s3 storage bucket is connected by external storage app. Syncing with Nextcloud Client PC folder directly to external storage S3. The problematic occurs when creating a folder on synced PC folder. This makes the folder and a ghost folder in subtree on S3

joshtrichards commented 10 months ago

This sounds like a configuration matter or, I suppose, possibly an issue with whichever client you're using.

We do all our testing primarily with MinIO so it's unlikely a bug like this would occur and make it to release (but anything is possible).

You've still failed to provide enough information to attempt to reproduce this. There are thousands of possible configuration combinations.

Please provide the configuration of your External Storage mount. Either a screen shot from your External Storage config section or the output of occ config:list files_external (or equivalent).

Please also provide screenshot from the MinIO Console Object Browser with "the folder and a ghost folder in subtree" that you're seeing.

EDIT: Oops I mean the output of occ files_external:list --output=json_pretty.

Section1977 commented 10 months ago

{
    "apps": {
        "files_external": {
            "enabled": "yes",
            "installed_version": "1.19.0",
            "types": "filesystem"
        }
    }
}

Section1977 commented 10 months ago

Screenshot_20231217_215734

joshtrichards commented 10 months ago

Sorry my mistake: Please send the output of occ files_external:list --output=json_pretty (I gave you the wrong command)

Section1977 commented 10 months ago

thanks for your help

[
    {
        "mount_id": 2,
        "mount_point": "\/test",
        "storage": "\\OCA\\Files_External\\Lib\\Storage\\AmazonS3",
        "authentication_type": "amazons3::accesskey",
        "configuration": {
            "bucket": "test",
            "hostname": "minio.server.lan",
            "port": "80",
            "region": "eu-central-1",
            "storageClass": "",
            "use_ssl": false,
            "use_path_style": true,
            "legacy_auth": false,
            "key": "secret",
            "secret": "secret"
        },
        "options": {
            "encrypt": true,
            "previews": true,
            "enable_sharing": false,
            "filesystem_check_changes": 1,
            "encoding_compatibility": false,
            "readonly": false
        },
        "applicable_users": [],
        "applicable_groups": []
    }
]

joshtrichards commented 10 months ago

Reproduced. Thanks for the report! :+1:

Only the folder placeholder appears to get created. I don't see duplicated content fortunately.

Did you notice this behavior after an upgrade? (If so it might help us figure out what version of NC introduced this regression).

Section1977 commented 10 months ago

if you download the content of the "empty" folder you get content :-) it is twice. never seen this before because I am new to S3

joshtrichards commented 10 months ago

Actually I think this is just the 0 byte placeholder object created by the S3 client to represent folders. Despite how the MinIO browser shows it, I see no indication of duplicate content. If you download the parent (in MinIO) there is no duplicated content.

Since S3 doesn't support folders natively, there are several ways of simulating the concept of folders in it. And, in turn, several ways of displaying in in S3 explorer clients. The core functionality is working appropriately here best as I can tell. This is merely a cosmetic misdirection within the MinIO browser. The extra folder is just the placeholder object that represents the folder.

Technically it only needs to exist until the folder has something else in it... In theory we could clear out the placeholder after the "folder" gets populated with files I guess... but that's not now AWS does it AFAIK. And we follow their convention. Also it would mean that if you delete the files in the folder, the folder would suddenly magically disappear. That wouldn't be very friendly IMO.

The reason this doesn't happen when creating a folder in the MinIO browser itself is because the approach they take seems to be to not really create a placeholder for a folder. Instead, if you create a "folder" in the browser and don't immediately put something in it, it never truly gets created. It doesn't look that way, but try to create a folder in the MinIO browser then go back to the parent of your bucket. The folder won't exist.

We actually create a placeholder.

Not familiar with pydio directly, but seems they suggest doing something similar if you want empty folders:

e.g. https://forum.pydio.com/t/how-to-create-a-folder-without-files-s3-postman/2293

nextcloud-command commented 9 months ago

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

Section1977 commented 9 months ago

s3 doesnt support folders so when the folder is empty you can automatically delete the folder and follow s3 convention. please make another solution for creating folders but not this way. it is more than irritating. pydio use hidden file to leqve folder but removing them if empty is also very useful.

nextcloud / server