versity / versitygw

versity s3 gateway
https://www.versity.com/products/versitygw/
Apache License 2.0
85 stars 15 forks source link

Posix backend: follow symlinks when listing buckets #644

Open kyleaupton opened 1 week ago

kyleaupton commented 1 week ago

Describe the solution you'd like I have a unique situation where all of the buckets I'd like to serve via the posix backend are symlinks to various locations on my linux machine.

Consider the following directory structure:

var/
└── s3-buckets/
    ├── bucket-1/  -->  /mnt/foo/bucket-1
    ├── bucket-2/  -->  /mnt/bar/bucket-2
    └── bucket-3/  -->  /mnt/baz/bucket-3

mnt/
├── foo/
│   └── bucket-1/
├── bar/
│   └── bucket-2/
└── baz/
    └── bucket-3/

Basically I'm trying to flatten out the things in /mnt. If I start versitygw with the parameters ./versitygw posix /var/s3-buckets nothing is returned when listing buckets. I believe that is because of the following:

https://github.com/versity/versitygw/blob/63c9e7503973ad2f49590cfc3303d894ff62ec2f/backend/posix/posix.go#L131-L135

After changing the above to the following it seems to return buckets and allow me to upload. (Far from an exhaustive test)

    for _, entry := range entries {
        fi, err := os.Stat(entry.Name())
        if err != nil {
            // skip entries returning errors
            continue
        }

        if !fi.IsDir() {
            // buckets must be a directory
            continue
        }

This works because os.Stat() will follow an items symlink unlike what comes back on a DirEntry item.

If there's another solution to how I can flatten out the things in /mnt without modifying the source structure I'm all ears. Interested to get y'alls thoughts. Thanks.

benmcclelland commented 1 week ago

I think we probably could make this change. I don't think we follow symlinks for listing objects within a bucket as well. I wonder if ew need an option to follow links, or if we should just enable this. I'll have to discuss with others what problems might arise from this.